<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Phase 3 — Auth and Cooperative Server — Arc Plan</title>
<style>
body { font-family: system-ui, sans-serif; max-width: 760px;
margin: 1.5em auto; padding: 0 1em; line-height: 1.5; color: #1a1a1a; }
h1 { font-size: 1.4em; }
h2 { font-size: 1.1em; margin-top: 1.4em; }
ol { margin: 0.3em 0; } li { margin: 0.25em 0; }
ol.slices { list-style: none; padding-left: 0; }
ol.items { list-style: none; padding-left: 0; }
.check { color: #1a7f37; font-weight: bold; }
.next { color: #b35900; font-weight: bold; }
.q { color: #8250df; }
code { background: #f2f2f2; padding: 0 0.25em; border-radius: 3px; }
em { color: #555; }
</style>
</head>
<body>
<h1>Phase 3 — Auth and Cooperative Server</h1>
<p style="margin:.8rem 2rem;padding:.7rem 1rem;background:#eef2f7">
<strong>Arc Post-Mortem Summary.</strong>
Started as authentication modernization and
grew a second thread after a production emergency showed the single-process
atto web server blocks every connection during long work (a huge mailbox
listing, an external IMAP probe, a password hash). It landed the immediate
server-stability fixes and then the cooperative non-blocking server, which
lets long work yield to the event loop and be re-entered, starting with IMAP.
The authentication rework proceeded alongside in the originally-listed
order.</p>
<p><em>Renamed from “Auth Modernisation” on 2026-06-19 once the
arc grew a second thread: a production emergency exposed that the
single-process atto web server blocks every connection whenever a request
does something long (a huge local-mailbox listing, an external IMAP probe,
a password hash). The stability fixes for those landed under “Server
stability work” below, and the next thread is the cooperative
non-blocking server that lets long work yield to the event loop and be
re-entered — starting with IMAP.</em></p>
<p><em>Arc started 2026-06-19; revised the same day after dropping TOTP.
Covers the v10 ToDo Phase 3 items <strong>in the order they were
originally listed</strong>: item 32 (modernise the image captcha) and
item 33 (replace the recovery-question system). Item 33(a) —
default new installs to
<code>EMAIL_RECOVERY</code> — and item 47 — the
database-backed session store — already landed.</em></p>
<h2>What exists today (verified 2026-06-19)</h2>
<ol>
<li>Two captcha modes already exist: <code>IMAGE_CAPTCHA</code>
(<code>CaptchaModel</code> draws letters with GD's built-in bitmap
font) and <code>HASH_CAPTCHA</code> — a proof-of-work mode
where the browser runs <code>hash_captcha.js</code> /
<code>sha1.js</code> to find a nonce meeting a difficulty
(<code>HASH_CAPTCHA_LEVEL</code>). The image mode is the one being
retired.</li>
<li>Recovery modes are <code>NO_RECOVERY</code> (0),
<code>EMAIL_RECOVERY</code> (1), <code>EMAIL_AND_QUESTIONS_RECOVERY</code>
(2); the default is already <code>EMAIL_RECOVERY</code> (33(a) done).
Recovery questions are driven by fixed
<code>register_view_recoveryN_*</code> locale keys.</li>
<li>Yioop already sends mail through the same paths the rest of v10
uses, so email one-time codes need no new transport — just a
short-lived signed/stored token and a verify step.</li>
</ol>
<h2>Plan (in the originally-listed order)</h2>
<ol class="items">
<li><span class="check">✓</span> <strong>Item 32 — retire
the image captcha; gate signup with proof-of-work, cheap invisible
checks, and an email confirmation.</strong> Image captchas are a
declining defence (machine solvers and captcha-farms beat them) and
a burden for blind users. Self-contained replacement, landed in
slices:
<ol>
<li><span class="check">✓</span> Auth + suggest forms
→ proof-of-work only: remove the global
<code>IMAGE_CAPTCHA</code> mode (the constant, the
<code>CAPTCHA_MODE</code> setting, the admin Captcha-Type
control, the <code>RegisterController</code> image branches, and
the now-dead image blocks in the register / recover / resend /
suggest views). <code>CaptchaModel</code> and
<code>setupGraphicalCaptchaViewData</code> stay alive for the
wiki placeholder, converted next. <em>This patch.</em>
<ul><li><span class="check">✓</span> Follow-up (folded in):
the resend-activation and recover-account forms each lost their
captcha, leaving the separator line above their button stranded
— removed that <code>border-top</code> on both (the
multi-field register / suggest forms keep theirs, where it still
reads as an action separator).</li></ul></li>
<li><span class="check">✓</span> Wiki forms →
proof-of-work automatically: <code>{{image-captcha}}</code> now
expands to a hidden field whose browser solves a proof-of-work
puzzle (no typing, no author-chosen type). A shared
<code>meetsProofOfWork</code> on the base controller does the
check; <code>setupGraphicalCaptchaViewData</code> became
<code>setupCaptchaViewData</code> (keeps the per-render word that
seeds the form's dedup hash, drops the image), and
<code>WikiElement</code> renders the proof-of-work inputs.
<code>SocialComponent</code> verification swaps the typed-word
match for the puzzle check while keeping the existing
new-row / reload / dedup behaviour. <code>CaptchaModel</code> and
the image plumbing are removed. <strong>Kept
<code>{{keyword-captcha}}</code></strong> — an author-chosen
keyword is a deliberate tool. <em>This patch.</em>
<ul><li><span class="check">✓</span> Refinement: dropped the
<code>{{image-captcha}}</code> syntax entirely (no legacy support)
and made the proof-of-work automatic on <strong>every</strong>
wiki form. Renamed to disambiguate — the
anti-bot check is a <em>proof-of-work</em> form check, not a
captcha: token <code>[{hash-captcha}]</code> →
<code>[{proof-of-work}]</code>, and
<code>setupCaptchaViewData</code> →
<code>setupProofOfWorkViewData</code>. One consistent pair does the
work: <code>WikiParser::proofOfWorkFormFields()</code> injects (from
<code>GroupModel::setPageName</code> when it wraps a form) and
<code>Controller::meetsProofOfWork()</code> checks. The proof-of-work
is always required; <code>{{keyword-captcha}}</code> is an
<strong>additional</strong> gate, so a form carrying one must pass
both. The injected answer field is skipped only when a keyword
captcha already supplies one (it shares the form's key column), so
there is never a duplicate field. Removed the dead
<code>system_component_image_captcha</code> /
<code>_hash_captcha</code> locale labels.</li></ul></li>
<li><span class="check">✓</span> Cheap invisible checks: a
honeypot field <strong>hidden from screen readers</strong> (off
screen, out of tab order, <code>aria-hidden</code>,
<code>autocomplete="off"</code>) is injected into every form by
<code>proofOfWorkFormFields</code>; <code>processWikiFormData</code>
rejects a submission that fills it, one that comes back faster than
<code>MIN_WIKI_FORM_DELAY</code> seconds, or one from an address
already inside the per-IP backoff (a new up-front
<code>getVisitor</code> gate reusing the same
<code>captcha_time_out</code> timeout the account forms use). All
three answer with the generic captcha-failed message and feed
<code>updateVisitor</code>, so a script cannot tell which check
caught it and repeat trips grow the timeout. <em>This patch.</em></li>
<li><span class="check">✓</span> Email confirmation before an
account activates (rides on Yioop's own mail; dovetails with item
33). The flow already exists — the
<code>email_registration</code> type adds the user inactive,
<code>sendActivationMail</code> sends a DKIM-signed message with a
verify link, and <code>emailVerification</code> activates the
account. Audited for deliverability and security; the findings are
being addressed in sequence:
<ol>
<li><span class="check">✓</span> The Date header used a
two-digit year (<code>DATE_RFC822</code>), a spam-filter signal
on every outbound message; now <code>DATE_RFC2822</code> in
<code>MimeMessage::build</code>.</li>
<li><span class="check">✓</span> The confirmation mail
reused the admin-activation subject; it now has its own
user-facing subject string.</li>
<li><span class="check">✓</span> The verify-link token was
compared with <code>==</code>; now <code>hash_equals</code> so
the check runs in constant time.</li>
<li><span class="check">✓</span> Resend re-sends the
activation mail to an inactive address with no throttle —
add a per-address / per-IP rate limit. Done: resendComplete
now throttles per requester IP and per hashed target email via
the VisitorModel timeout mechanism, so neither one IP nor one
inbox can be flooded.</li>
<li><span class="check">✓</span> The verify URL carries
the email address in plain text — replace it with an
opaque token so no address rides in the link. Done: the link
now carries a signed token holding only the user id (and an
expiry), resolved by UserModel::getUserById; no email travels
in the URL.</li>
<li><span class="check">✓</span> The activation link never
expires — give the token an expiry (the short-lived token
store item 33 wants). Done: the token now signs an expiry time
alongside the user id and parseActivationToken rejects a stale
or expiry-tampered link; lifetime is C\ACTIVATION_LINK_TIMEOUT
(a stateless self-expiring token, the no-row option 33(b)
weighs).</li>
<li><span class="check">✓</span> Verify and resend return
account-existence-revealing
messages — make them generic. The resend form took a
username and answered with three different messages (no such
account / already active / mail sent), so anyone could type
usernames at the public form and learn from the reply alone
which accounts exist and whether they are active —
account enumeration. Fix (done): resendComplete now returns
one neutral message in every case and only actually re-sends
when a real, still-inactive, non-throttled account matches;
the verify side keeps its helpful "already activated" message
since that only shows to someone holding a valid signed
link.</li>
<li><span class="check">✓</span> Optional polish: the
mail carried no <code>Reply-To</code> and was plain text.
Done: activation mail now sets a <code>Reply-To</code> from
the new <code>MAIL_REPLY_TO</code> config (defaults to the
sender mailbox, can be pointed at a staffed support address),
and the body got a light HTML face lift. It now goes out as
<code>multipart/alternative</code> — a plain-text part
(kept for text-only mail apps and the spam filters that
expect one) and a light HTML part with the activation and
unsubscribe links made clickable and the user-supplied names
and URLs escaped — assembled by a new
<code>MimeMessage::alternativeBody</code> helper and carried
through by letting <code>MimeMessage::build</code> honor a
caller-supplied Content-Type.</li>
</ol></li>
<li>Update the Wiki Syntax document to match (image-captcha gone,
keyword-captcha kept, proof-of-work applied automatically).</li>
</ol></li>
<li><strong>Item 33 — replace the recovery-question system.</strong>
<ol type="a">
<li><span class="check">✓</span> Default new installs to
<code>EMAIL_RECOVERY</code> — already done.</li>
<li><span class="check">✓</span> Email-based one-time login codes: generate a short,
time-limited, single-use code (or signed link), mail it to the
registered address, and verify it to sign the member in or let
them recover, without a saved password and without any external
app. Needs a short-lived token store (a small table or signed
token with an expiry), a request step, a verify step, and rate
limiting so the mailbox can't be hammered.
<br><em>Built:</em> a single-use, table-backed
code in the new <code>LOGIN_CODE</code> table (migration
<code>upgradeDatabaseVersion113</code>, <code>DATABASE_VERSION</code>
now 113), 8 chars from an unambiguous alphabet, 15-minute lifetime
(<code>LOGIN_CODE_TIMEOUT</code>), at most 5 wrong guesses
(<code>LOGIN_CODE_MAX_TRIES</code>). Storage lives on
<code>SigninModel</code>; the controller (<code>loginCode</code>,
<code>processLoginData</code>, <code>loginCodeComplete</code>)
emails the code <em>and</em> a one-click button carrying a signed
token plus the code, both verifying the same stored hash and
signing the member straight in. The request form is reached from a
new link on the sign-in page and is shown by a method on
<code>SigninView</code>. Per-IP and per-email
<code>ONE_MINUTE</code> throttling and a generic anti-enumeration
message mirror the resend flow. Covered by
<code>SigninModelTest</code>.</li>
<li><span class="check">✓</span> For an offline / airgapped install that keeps a question
mode, let users type their own question and answer; drop the
rigid <code>register_view_recoveryN_*</code> translation
infrastructure and remove the
<code>EMAIL_AND_QUESTIONS_RECOVERY</code> mode.</li>
</ol></li>
</ol>
<h2>Server stability work (now in-arc)</h2>
<ol>
<li><span class="check">✓</span> Atto server hang hotfix
(<code>WebSite.php</code>): a peer whose TLS faulted mid-stream (a
“bad record mac” alert) left its socket perpetually
readable, so the event loop spun at full CPU and
<code>cullDeadStreams</code> never reaped it (the read path kept
refreshing its activity time), which eventually blocked all new
connections including the command-line restart. Fix: in
<code>parseH1Request</code>, a readable socket whose read returns
<code>false</code> (TLS fault) or empty at end-of-file is now torn
down at once with <code>shutdownHttpStream</code>; the read's warning
suppression also swallows the <code>SSL operation failed</code> notice
(same client-side transport class as the existing “reset by
peer”); and the activity-time refresh in
<code>processRequestStreams</code> is guarded so a connection closed
during the read leaves no stray timer.</li>
<li><span class="check">✓</span> Fuller hardening (same file):
the read path no longer refreshes a connection's activity time on a
zero-byte readable pass. A new <code>read_made_progress</code> flag is
reset before each connection is handled and lowered by the H1 read
when a readable socket returns no bytes yet is neither at end-of-file
nor errored (a stalled or partial-record peer); only when the read
actually pulled bytes does <code>processRequestStreams</code> refresh
the time. So even a peer that stays readable while delivering nothing
ages out on the idle timeout instead of holding the loop. Other
protocols leave the flag true, so their behaviour is unchanged.</li>
<li><span class="check">✓</span> MailSite audit + matching fix
(<code>MailSite.php</code>): checked the mail server for the same
class of bug. The stalled-TLS-handshake case is already safe —
handshakes are non-blocking with a 10-second deadline, watched
read-only (so a connected, always-writable socket cannot spin the
loop), and <code>cullDeadStreams</code> reaps any handshake past its
deadline every tick (select is capped at 5s). The idle path was also
already safe (it stamps the activity time only after a non-empty
read), and the write path already closes on a failed write. The one
gap, the same one fixed in <code>WebSite.php</code>: an established
secure connection whose read faulted (a <code>false</code> return)
just returned and spun until the 30-minute idle cull. Fixed
<code>readClient</code> to close at once on a read fault or an
end-of-file read, matching the web server.</li>
<li><span class="check">✓</span> Web mail-check hang fix
(<code>ImapClient.php</code>): checking mail in the web UI could hang
the whole site. The STARTTLS path called
<code>stream_socket_enable_crypto</code> on a blocking socket, which
ignores the read timeout — so an IMAP peer that accepted
STARTTLS and then stalled mid-handshake blocked the single-process web
server forever, and no other web request could be served. Added
<code>enableCryptoBounded</code>, which runs the handshake
non-blocking and waits only up to the connect timeout
(<code>MAIL_IMAP_CONNECT_TIMEOUT</code>, 15s) before giving up;
<code>connectStartTls</code> now uses it. The implicit-TLS path
(<code>connectImaps</code>) already bounds its handshake through the
<code>stream_socket_client</code> connect timeout. (<code>SmtpClient</code>
has the same blocking-handshake pattern but runs in the background mail
sender, not a web request, so it is a lower-priority follow-up.)</li>
<li><span class="check">✓</span> Huge local-mailbox inbox freeze
fix (<code>MailSiteMailBackend::listMessages</code>): opening a large
MailSite (local) inbox in the web UI froze the whole single-process web
server, blocking every other connection. The listing shaped
<em>every</em> message in the folder — reading one header block per
message — before paging down to one window, so a 90,000-message
INBOX did ~90,000 small file reads per window request. An SSD hid the
cost (the first window returned, slowly); a spinning disk turned it into
minutes of seeks that hung the server, and later windows failed. Changed
to window-then-shape: the date sort, the unread/flagged filters and the
windowing now run on the lightweight index records (which already carry
uid, date and flags), and message files are read only for the ~25
messages on the returned window. Added
<code>listMessagesWindowReadsOnlyWindowTestCase</code> (a counting
MailSite subclass) asserting that listing one window of a 60-message
folder reads at most one window's worth of message headers, so a return
to shaping the whole folder is caught.</li>
<li><span class="check">✓</span> Unread-badge login freeze fix
(<code>MailUnreadProbe</code>, <code>Component</code>,
<code>SocialComponent::userMailBaseData</code>): the unread-mail badge
in the shared page chrome was computed by connecting to every external
IMAP account (connect + LOGIN + STATUS) on a cache miss, inside the
single-process web request. So one user logging in (or any page view
after the 300s cache expired) with a slow or unreachable external
account froze the whole server for every other visitor. Split the badge
into a cache-only read used by the page chrome
(<code>MailUnreadProbe::cachedCount</code>, which never opens a
connection and returns the last cached value or 0) and the existing
computing <code>count()</code>, now called only from the Mail activity
(<code>userMailBaseData</code>), where talking to the mail servers is
expected and the listing already connects. General browsing and login
no longer wait on mail; the badge catches up the next time the user
opens their mail. (Making even the Mail activity's external-IMAP calls
non-blocking is the separate async-IMAP architectural follow-up.)</li>
</ol>
<h2>Cooperative non-blocking server</h2>
<ol class="slices">
<li><span class="check">✓</span> <strong>Slice 1 —
cooperative scheduler.</strong> A CooperativeScheduler inside the
WebSite atto file drives tasks a little at a time, interleaved, each
with a completion callback (its result or the exception it threw) and a
30-second work-time budget; WebSite holds one lazily and steps it each
loop pass. Under Apache the work runs straight through. Back-ports to
the atto project.</li>
<li><span class="check">✓</span> <strong>Slice 1b — use
fibers, not generators.</strong> The scheduler drives <code>\Fiber</code>
tasks so a deep mail-socket read can suspend the whole request without
rewriting the controller chain above it; <code>deferTask</code> takes a
callable, wraps it in a fiber under atto, runs it inline under
Apache.</li>
<li><span class="check">✓</span> <strong>Slice 1c — route
opt-in and deferred response.</strong>
A route opts into running its whole handler inside a fiber that may
suspend mid-render. Each such request's environment (the superglobals
plus <code>header_data</code>/<code>content_type</code>/
<code>current_session</code>) is swapped around every resume and its
output captured per resume; the response is built and sent only when
the fiber finishes (a 500 if it throws or runs over budget). Wired for
HTTP/1.1, HTTP/2, and HTTP/3 (the last through <code>H3Listener</code>);
the H1 round trip is core-tested in the sandbox, the live protocols
need testing on the server.</li>
<li><span class="check">✓</span> <strong>Slice 2 —
fiber-suspending <code>ImapClient</code>.</strong> Reads
(<code>readLine</code>/<code>readBytes</code>) and the bounded TLS
handshake now hand the loop back with
<code>Fiber::suspend(['read' => $sock])</code> at a would-block
point when inside a fiber, and block exactly as before otherwise (so
Apache and non-cooperative callers are unchanged); the loop polls
in-flight tasks at a short interval (<code>COOPERATIVE_POLL</code>)
rather than spinning. A socket-pair test proves a read suspends until
the server answers. (Wiring the wait sockets straight into the loop's
select to drop the poll is a possible later refinement; writes block
briefly as now.)</li>
<li><span class="check">✓</span> <strong>Slice 3 — route web
IMAP through the cooperative path.</strong> Web mail requests
(<code>a=userMail</code>, bar attachment downloads) now run inside a
deferred fiber, so the folder and message listing, message view, and
the unread-count probe — everything that reads IMAP — yield
to the loop while a slow or down mail server is being waited on rather
than stalling every other connection. For this, <code>deferResponse</code>
now treats a <code>webExit()</code> (redirect or normal early exit) as
a clean finish, the way the synchronous path does, instead of a 500.
Under Apache the requests run straight through unchanged.</li>
<li><span class="check">✓</span> <strong>Slice 4 — login
password hash via a short-lived worker.</strong> The bcrypt hash
(<code>crawlCrypt</code>, cost 12, a good fraction of a second) cannot
be split, so when it runs inside a fiber it is handed to a fresh
short-lived PHP process (started with <code>proc_open(PHP_BINARY...)</code>;
chosen over forking the live event-loop server, which would inherit
its sockets and is not portable). The password and salt go to the
helper over its stdin (never the command line), and the login task
<code>Fiber::suspend(['read' => $pipe])</code>s until the helper
writes the hash back. Outside a fiber <code>crawlCrypt</code> is the
plain crypt it always was, so every other caller and Apache are
unchanged. Signin requests (those carrying <code>u</code> and
<code>p</code>) are now deferred so the hash actually runs in a fiber.
A test proves the helper produces the identical hash and the fiber
suspends on its pipe.</li>
<li><span class="check">✓</span> <strong>Slice 5 —
cooperative media-list assembly.</strong> A wiki page of type Media
List with a thousand or more resources hung the server while its list
was built. Two fixes: the per-resource has-thumbnail check was a linear
<code>in_array</code> over every thumbnail, making the whole loop grow
with the square of the count — it is now an O(1) keyed lookup; and
the assembly loop in <code>GroupModel</code> gives the loop a turn every
<code>RESOURCE_LIST_YIELD</code> resources (via
<code>cooperativeYield()</code>, a no-op outside a fiber), with wiki
requests now deferred so that yield takes effect. To keep that yield
free, the scheduler now tells the loop when it has immediate work (a
task that suspended plainly, not on a socket), so a CPU- or disk-bound
job is resumed at once rather than waiting out the poll interval, while
socket waits are still polled gently.</li>
<li><span class="check">✓</span> <strong>Slice 6 —
cooperative <code>SmtpClient</code>.</strong> Its response read used a
blocking <code>fgets</code> and its STARTTLS handshake a single
blocking call, both of which froze the loop waiting on the mail
server. The reader is now buffered the way <code>ImapClient</code>'s
is and waits for the socket to be readable (suspending the fiber when
cooperative, blocking exactly as before otherwise), and the STARTTLS
handshake uses the same bounded, fiber-yielding upgrade — which
also fixes its TLS version to 1.2/1.3 only, dropping the obsolete bare
method that allowed TLS 1.0/1.1. The web requests that send mail
(registration, group feeds and group management; <code>userMail</code>
was already deferred in Slice 3) are now deferred so those sends run in
a fiber. A socket-pair test proves a response read suspends until the
server answers. The plain (non-implicit-TLS) TCP connect is now
cooperative too: in a fiber it asks for a non-blocking connect and
suspends until the socket reports it has connected, so a slow or
unreachable server no longer freezes the loop during the connect
either (tested against a local listener); outside a fiber it is the
same blocking connect as before. The seconds-to-microseconds
conversion the bounded handshake and the loop clamp use is now the
named constant <code>MICROSECONDS_PER_SECOND</code> rather than a bare
1000000. (Two blocking points remain, both rare and bounded by the
connect timeout: the implicit-TLS, port-465 connect, whose handshake
is part of the connect; and the IMAP client's connect, which can take
the same async treatment next.)</li>
<li><span class="check">✓</span> <strong>Slice 6b —
<code>sendImmediate</code> moved to a worker process.</strong>
Reconsidered the SMTP approach: rather than make each socket step in
the web process cooperative, a send done inside a fiber now hands the
<em>whole</em> exchange to a short-lived helper process — the
same way the password hash gets off the loop — and suspends on
the helper's pipe until it answers. The connect, the TLS handshake,
and the conversation all happen in the helper, so none of them can
block the loop, and the helper's success flag and log are copied back
onto the client. The per-I/O suspend code added in Slice 6 (the
buffered reader, the bounded handshake, the async connect) is removed
as no longer needed; the TLS-1.2/1.3 fix is kept. Bulk mail already
goes through <code>sendQueue</code> and the background updater, so only
the occasional <code>sendImmediate</code> reaches a helper. A test
drives a send in a fiber against a closed port and confirms it
suspends on the helper pipe and carries the failure back. The helper
discards any stray warning markup so only its JSON result reaches the
pipe.</li>
<li><span class="check">✓</span> <strong>Slice 7 —
cooperative read locks on the local <code>FileMailStorage</code>.</strong>
This closes the freeze the whole arc was opened for. Clicking the
local inbox runs, in the web process,
<code>MailSiteMailBackend->listMessages</code> → in-process
<code>FileMailStorage</code>, which takes a shared <code>flock</code>
on the folder index while the mail daemon may hold it exclusively;
Slice 3 put that request in a fiber, but a synchronous
<code>flock</code> cannot suspend, so the loop froze. The four
read-path shared locks (<code>readFolderIndex</code>,
<code>readReuidJournal</code>, <code>readFlagsSnapshot</code>,
<code>liveUidsFromIndex</code>) now go through a
<code>cooperativeFlock</code> helper: inside a fiber it asks for the
lock without blocking and, if it is held, hands the loop a turn and
retries; outside a fiber (the daemon, the command-line tools) it is a
plain blocking lock, unchanged. The local backend now yields like the
IMAP backend already did. Tested: a shared-lock read started in a fiber
suspends while the file is held exclusively and acquires the moment it
frees.</li>
</ol>
<h2>Follow-on arc (B)</h2>
<p>The MailSite <em>daemon's</em> own <code>listen()</code> loop (a
separate process, the same single-process <code>stream_select</code> shape
WebSite had) still blocks one mail connection on another. Making it
cooperative is a much bigger surface and raises the scheduler question
(<code>CooperativeScheduler</code> lives in WebSite.php and atto files are
self-contained, so the daemon would get its own copy of the step logic
rather than reaching across files). That work, plus auditing the rest of
the atto servers for the same blocking pattern and writing worked examples
of the new fiber primitives, is its own arc, started right after this
patch.</p>
<h2>Other work</h2>
<ol>
<li><span class="check">✓</span> <strong>Deferred HTTP/2 send to
a closed connection.</strong> When a client disconnected while a
deferred response was still pending, <code>emitDeferredH2</code>
fetched a now-null connection object and <code>sendH2Body</code>
faulted taking a reference to <code>protocol_state</code> on null
(uncaught Error in the cooperative loop). It also queued the
response headers first, orphaning those bytes in the out-stream
buffer for a connection that would never drain — a slow leak
that can feed the out-of-memory crashes. <code>emitDeferredH2</code>
now returns as soon as it sees the connection is gone (before
queuing anything), and <code>sendH2Body</code> has a matching
defensive guard.</li>
<li><span class="check">✓</span> <strong>MailSite.php long-line
cleanup.</strong> Re-wrapped the 117 over-80-column lines flagged
when the IMAP partial-fetch fix landed; all were docblock comment
lines, so only wording wrapped — no code changed.</li>
<li><span class="check">✓</span> <strong>Manage Users table
overflow.</strong> A long email address widened the user-listing
table past the activity border, because the cells used
<code>word-wrap: break-word</code>, which does not let a long
unbroken token shrink the column. The cells now use
<code>overflow-wrap: anywhere</code> so such tokens break, and the
table is capped at <code>max-width: 100%</code> so it can never
exceed its container.</li>
<li><span class="check">✓</span> <strong>IMAP partial fetch
(<code>BODY[...]<offset.count></code>).</strong> Apple Mail kept
re-requesting the same message in a loop: it asked for a byte range
with <code>UID FETCH ... BODY.PEEK[TEXT]<4181.1215></code>, but
the server ignored the range and returned the whole section with no
origin-octet marker, so the client rejected the reply and retried
forever (and never showed the activation mail). The FETCH-item parser
now reads the <code><first.count></code> suffix and the renderer
returns just that slice, labelled <code>BODY[TEXT]<first></code>
as RFC 3501 7.4.2 requires; a count past the end yields the shorter
remainder and non-partial fetches are unchanged.</li>
<li><span class="check">✓</span> <strong>CodeTool unit colour.</strong>
The <code>[x / y] passed</code> summary on each test method now prints
green when all of that method's tests passed and red when any failed.
The colour is added only when output is an interactive terminal (a log
or pipe stays plain), and Windows VT100 support is switched on first so
the same codes work on both Windows and Unix-like terminals.</li>
<li><span class="check">✓</span> <strong>HTTP/3 streaming
methods.</strong> <code>H3Listener</code> called
<code>$site->setStreamingProtocol()</code> and
<code>takeStreamingProducer()</code>, which never existed on
<code>WebSite</code> (the H3 request path would have fatalled).
Implemented both, and let <code>stream()</code> park a producer for
<code>h3</code> as it already did for <code>h2</code>, so the HTTP/3
path can run as intended.</li>
<li><span class="check">✓</span> <strong>tl() locale
extraction audit.</strong> Strings in views/elements/controllers
were rendering as raw keys on pollett.org because the locale
extractor only captures literal <code>tl("key")</code> calls. Fixed
every place a key reached <code>tl()</code> through a variable, a
concatenation, or a nested call. <code>MailElement</code>'s header
and DKIM-detail row helpers now translate at the call site
(<code>tl("literal")</code>) rather than inside the helper; the
scheduled-status and DKIM-summary labels use explicit per-value
<code>tl()</code> branches instead of
<code>tl("prefix_" . $status)</code>; the IMAP/SMTP TLS-mode
dropdowns pre-translate their option arrays; and
<code>StoreComponent</code>'s doubled <code>tl(tl(...))</code> (the
bogus "tl" missing string) became a single call. Added the four
genuinely-untranslated keys
(<code>machinestatus_view_no_queue_server</code>,
<code>machinestatus_view_no_fetchers</code>,
<code>accountaccess_component_activityname_doesnt_exists</code>,
<code>accountaccess_component_modifier_doesnt_exists</code>) to
<code>configure.ini</code>. Verified by re-running the extractor's
own regex against the touched files: zero bogus captures, and every
previously-pruned key now extracts as a literal. Rendered output is
byte-identical.</li>
<li><span class="check">✓</span> <strong>Remote images in
HTML mail: defer-and-reveal, no proxy.</strong>
Blocked remote images are not stripped or swapped. The
sanitiser leaves <code>src</code> intact and adds
<code>loading="lazy"</code> plus a
<code>mail-blocked-image</code> class
(<code>display:none</code>): a lazy image with no layout box
is never near the viewport, so the browser does not fetch it
and no tracker pixel fires on render. The "load images" button
removes the class; each revealed image then loads natively,
directly from its host. This went through a same-origin proxy
first (every image, then narrowed to an <code>onerror</code>
fallback for the few hosts that send a
Cross-Origin-Resource-Policy header and so refuse cross-origin
embedding), but the proxy was removed entirely: its server-side
fetch is a blocking <code>curl_exec</code> in
<code>FetchUrl::getPage</code>, and atto is a single-process
cooperative server, so each proxied image stalls the whole
event loop for the length of the remote fetch. Over HTTP/2 the
page document and every other resource multiplex on one
connection, so a single blocked fetch truncates the in-flight
page itself (<code>NS_ERROR_NET_PARTIAL_TRANSFER</code>) and
aborts queued requests. Direct loading is the rule and the only
rule. The cost: a handful of CORP-locked images (the sign-in
mail's logo among them) show broken after "load images"; every
other host loads fast. Restoring those would require a
cooperative (yielding) fetch on atto rather than a blocking
one -- deferred as future work, not built here.</li>
<li><span class="check">✓</span> <strong>Cooperative IMAP
body reads: yield during a large transfer.</strong> A webmail
request already runs inside a cooperative fiber (index.php routes
userMail through <code>WebSite::deferResponse</code>), and
<code>ImapClient</code>'s reads already suspend through
<code>waitReadable()</code> -- but only when the socket would
block. A mail server streaming a large message body keeps the
socket continuously readable, so <code>readBytes()</code> read the
whole literal in one tight 4096-byte loop that never yielded,
monopolising the single-process server for the length of the
transfer. On a heavy message (Peacock/Zillow) that froze the whole
site while the body came in, and starved the sibling resource
requests on the same HTTP/2 connection -- which is why
mailmessages.js could arrive truncated and the "load images"
button came up dead until a reload. <code>readBytes()</code> now
counts bytes pulled from the socket and, once past a
<code>COOPERATIVE_READ_YIELD</code> (64 KB) run, suspends the
fiber for one loop pass before continuing, so every other
connection makes progress during a big fetch. Small messages read
straight through with no yield; outside a fiber (Apache) the
<code>Fiber::getCurrent()</code> guard skips it and the read blocks
as before. Verified by driving a 293 KB read inside a fiber
over a socket pair: it yields several times and returns the
complete payload. The DKIM verify's <code>dns_get_record</code> is
a separate synchronous blocker on the same view path and is left
for follow-up (a blocking resolver cannot be made to yield without
an async DNS path).</li>
<li><span class="check">✓</span> <strong>Open a message
without a full page navigation.</strong> Clicking an inbox subject
used to be an ordinary link navigation: the browser threw away the
loaded page and refetched, reparsed, and re-ran the whole Mail
shell (every script, style, and init) just to show one message --
slow, and on a heavy message it was during that reload that the
blocking IMAP read (fixed above) starved sibling resources. The
subject links now carry a <code>mail-open-message</code> class; a
new <code>initMailMessageNav</code> intercepts a plain left click
on one, fetches that same URL, lifts the
<code>.account-messages</code> content out of the returned page,
swaps it into the current pane, and re-runs
<code>initMailMessageView</code> to wire the new view's controls.
The shell, its scripts, and its styles stay put, so opening a
message is now an in-place swap rather than a reload. Only the
three message-view controls need re-wiring (raw toggle, load
images, DKIM badge); reply / forward / back are ordinary links and
move / delete are self-contained forms with inline handlers, so
nothing else is stranded by the swap. Modifier and middle clicks
are left alone so open-in-new-tab still works;
<code>history.pushState</code> keeps reload and the back button
honest (a back navigation reloads whatever the address bar then
names); and any failure -- a network error or a response without
the pane, such as a re-login redirect -- falls back to an ordinary
navigation so a click is never lost. The server still renders the
full page and only its message pane is used; a dedicated fragment
response would save that extra render and is a sensible later
optimisation. Going back to the inbox stays a normal navigation, so
the inbox's own many handlers re-wire the ordinary way.</li>
<li><span class="check">✓</span> <strong>Load-images button:
delegate the click and force the load.</strong> After the in-place
open, the load-images button came up with no click listener -- only
a full page reload wired it -- so re-wiring it per element at swap
time was unreliable. The handler is now bound once on the document
(<code>initMailLoadImages</code>) and matched by
<code>closest('.mail-message-load-images-button')</code>, so it
fires for the button no matter how or when the message view enters
the page, with no dependence on a re-init running at the right
moment. The reveal is also made deterministic: as well as dropping
the <code>mail-blocked-image</code> class it removes
<code>loading="lazy"</code> and re-assigns each image's
<code>src</code>, which starts the fetch even for an image that
arrived through <code>innerHTML</code> -- where a still-hidden lazy
image otherwise never begins loading, the case that left "load
images" looking dead until a reload.</li>
<li><span class="check">✓</span> <strong>Show a loading
indicator while a message is fetched.</strong> The in-place open
fetches in the background, so a click gave no feedback at all until
the message appeared -- the browser's own page-load indicator no
longer applies. The reading pane is now filled with a small
spinner and a translated "Loading" label
(<code>mail_element_loading</code>, passed to the script as
<code>window.MAIL_LOADING</code> the same way the triangle glyphs
are) the moment the fetch starts, and the message replaces it on
arrival. The spinner is pure CSS and honours
<code>prefers-reduced-motion</code>.</li>
<li><span class="check">✓</span> <strong>Fix invalid HTML in
the folder-ops strings.</strong> The folder-delete confirmation
text carried literal double quotes around its <code>%s</code>, and
it is emitted into a double-quoted
<code>data-mail-element-folder-delete-confirm</code> attribute, so
the quotes closed the attribute early and the page no longer
validated. The other strings in that span carry no quotes, matching
the span's plain-attribute convention; the confirmation string is
brought into line by using single quotes around the folder name,
which leaves <code>%s</code> intact for the script's substitution
and needs no change in the view or the script.</li>
<li><span class="check">✓</span> <strong>Make the resize
handle on the right of the favorite column work.</strong> The
favorite (star) column had a handle on its left (the subject
column's resizer) but only a plain border on its right, which
looked like a handle yet did nothing. A real
<code>mail-col-resizer</code> is now placed in the star header
cell (resizing the star column against From), and the redundant
cell border it replaces is removed so the boundary is drawn once.
The drag clamp is also corrected: it previously refused any move
while a column was under the minimum width, which froze a handle
next to the narrow star column once the list pane was wide; it now
blocks only a column that is actively shrinking past the minimum,
so an already-narrow column can still be widened.</li>
<li><span class="check">✓</span> <strong>Permit safe inline
CSS in message bodies.</strong> The sanitizer used to strip every
style attribute, so a formatted newsletter rendered unstyled and,
worse, its preheader (a line meant to be hidden with
<code>display:none</code>) showed at the top and fixed desktop
widths read poorly on a phone. Inline style is now filtered rather
than dropped: a curated property allowlist (colour, typography,
spacing, borders, lists, display, visibility) is kept, while
anything that can lift content over the page (position, z-index,
float) is dropped, any value carrying a hazard
(<code>url()</code>, which would beacon a tracker and bypass the
remote-image block, plus <code>expression()</code>,
<code>@import</code>, and escape tricks) is dropped, and a
fixed-pixel <code>width</code> is dropped so a message stays fluid
in a narrow pane (a percentage or <code>max-width</code> is kept).
Because a blocked image may now carry its own
<code>display:block</code>, the sanitizer strips display and
visibility from blocked images and the hiding rule is marked
important, so a tracker image cannot show -- and therefore load --
through its own style. New unit tests cover the kept/dropped split,
the width filter, and the blocked-image display strip; a stale test
still asserting the superseded <code>data-mail-src</code> blocking
contract was corrected to the current lazy-plus-class one.</li>
<li><span class="check">✓</span> <strong>Stop wide message
tables overflowing the pane.</strong> With the fixed pixel widths
now dropped, a newsletter's layout tables fall back to automatic
width, which sizes a table to its longest single line rather than
wrapping -- so a centred 24px heading pushed the body well past a
phone's width. Message-body tables are now capped at the width of
their container (<code>.mail-message-body-html table</code> gets
<code>max-width: 100%</code>, matching the existing rule on
images), so an over-wide table wraps to fit. A wide desktop pane is
unaffected: the cap only binds when a table would otherwise exceed
the pane, so a 600px newsletter still renders at its natural
width.</li>
<li><span class="check">✓</span> <strong>Stop the app's
paragraph minimum width leaking into a message.</strong> The cap
above was not enough on a phone: the chrome's own
<code>.mobile p</code> rule gives every paragraph a 300px minimum
width, and that rule also matches the paragraphs inside a rendered
message. In a two-column newsletter each column then demanded
300px, so the layout table could not shrink below ~600px and the
body overflowed regardless of the max-width cap (an intrinsic
minimum beats a maximum). The minimum is now reset to zero for
message paragraphs and table cells
(<code>.mail-message-body-html</code> p/td/th/table), so the
columns collapse and the text wraps; because mail.css loads after
search.css the equal-specificity reset wins.</li>
<li><span class="check">✓</span> <strong>Keep a loaded image
inside the pane.</strong> A message fit once its remote images were
blocked, but loading them widened it again: a banner image carried
an inline <code>max-width: 700px</code> together with a
<code>width="600"</code> attribute, and an inline maximum outranks
the rendering rule's <code>max-width: 100%</code>, so it drew at
600px and overflowed a phone. The image cap on
<code>.mail-message-body-html img</code> is now marked important so
the responsive cap (and <code>height: auto</code> with it) wins
over whatever width the message sets inline; an image can never
exceed its column and stays in proportion.</li>
<li><span class="check">✓</span> <strong>Keep the DKIM detail
on screen on a phone.</strong> The signature-detail panel hangs off
the green badge, anchored to its right edge, and the badge sits 64px
in from the header's right edge; on a narrow screen a 30em panel
then ran off the left and the row labels were cut off. On mobile the
panel is now pinned to the viewport as a full-width sheet along the
bottom and scrolls within itself when it is taller than the screen,
so every label and value is reachable.</li>
<li><span class="check">✓</span> <strong>Let the back link
read correctly right to left.</strong> The folder back link drew a
hardcoded <code>«</code> arrow in the view, outside the
translated string, then appended the translated "Back to %s". A
right-to-left locale could never move or mirror that arrow, because
the directional glyph lived in the markup rather than in text a
translator owns. The arrow now lives inside the translated string
and the "Back to" wording is dropped, so the English value is
"« %s" and the link reads "« Inbox"; a
right-to-left locale can place the arrow on the trailing side in its
own value. Only the English value is changed here; other locales
keep their wording until retranslated.</li>
<li><span class="check">✓</span> <strong>Give the Raw button
and signature badge their own line on a phone.</strong> Both float
in the top-right corner of the header box, which is fine on a wide
pane but overlapped the Subject on a narrow screen. On mobile they
now sit in the normal flow on a single line just above the Subject,
so nothing covers the subject text; the desktop corner placement is
left unchanged.</li>
<li><span class="check">✓</span> <strong>ConfigureTool
upgrade-locales command.</strong> During ongoing work, seeing edited
help pages and resource locale files in a configured work directory
meant bumping the resources-wiki version so the normal startup check
would push them out. The configuration tool now takes an
<code>upgrade-locales</code> command that force-pushes the locale
strings together with the version-gated wiki and help pages and
resource files from the source tree into the work directory,
regardless of the stored version. The library
<code>upgradeLocales()</code> gained a force argument that drives
that unconditional push; its normal version-gated behavior on
startup is unchanged.</li>
<li><span class="check">✓</span> <strong>Upgrade Locales as a
menu step.</strong> The force-push is now item (4) Upgrade Locales
on the configuration tool's main menu, between Set Default Locale
and Debug Display Set-up. The menu method does the push and reports
its outcome through the usual menu message; the upgrade-locales
command runs that same menu method and prints the message it sets,
so the command reuses the menu step's action rather than carrying
its own copy, the way the other commands reuse their step.</li>
<li><span class="check">✓</span> <strong>Keep valid locales
loading after the missing-locale guard.</strong> The new guard in
the locale initializer tested the writing mode as truthy rather than
empty, so every real locale row took the missing-locale branch: the
locale name was dropped and the writing mode was forced to
left-to-right, which also flattened right-to-left locales. The test
now checks for an empty writing mode, so a present locale keeps its
name and direction (for example the Persian locale loads as
right-to-left again) while a row with no usable data still falls back
to the defaults.</li>
<li><span class="check">✓</span> <strong>Hotfix: upgrade no
longer fatals on the DKIM key step.</strong> Upgrading an older
install ran the version-101 database step, which calls
<code>DkimKey::ensureKeyPair()</code>, but the version-functions file
sits in the library namespace and never imported the class, so the
bare name resolved to a nonexistent library-namespace class and threw
a fatal class-not-found. It now imports the mail
<code>DkimKey</code> class, the same way the profile model and the
bulk-email job do. The step was latent in development because a
database already at the current version never runs it.</li>
<li><span class="check">✓</span> <strong>Hotfix: swept the
upgrade code for unimported mail classes.</strong> The next upgrade
step fataled the same way on <code>MailSiteFactory</code>, so rather
than fix it alone the whole library-root was scanned (tokenized, so
string literals and docblock mentions are not false positives) for
any class used as <code>X::</code> or <code>new X</code> that lives
in a library sub-namespace yet is neither imported nor defined in the
library root. The only real hit besides the already-fixed
<code>DkimKey</code> was <code>MailSiteFactory</code>; both are now
imported, and the sweep reports nothing else outstanding.</li>
<li><span class="check">✓</span> <strong>Trending-term regex:
escape feed terms and revive possessive matching.</strong> When
counting trending terms, the feed term was interpolated straight into
a regular expression, so a term carrying a backslash sequence (for
example a stray <code>\g</code> or <code>\u</code> from feed text)
made PCRE2 fail to compile the pattern and logged a warning on every
such term. Each of the four match sites now passes the term through
<code>preg_quote</code> so it is treated as literal text. The same
pass fixed the long-dead possessive branch: a word like
<code>engineering_pos_s</code> (the tokenizer's marker for
"engineering's") never matched because the guard compared against a
back-slashed <code>"\_pos\_s"</code> instead of the six-character
<code>_pos_s</code> marker, then built its pattern from the marker
rather than the word, and required whitespace after the apostrophe
that "engineering's" does not have. It now strips the marker, builds
the pattern from the word, and allows the apostrophe to be followed
directly by the trailing letters, so possessives match in plain
(<code>engineering's</code>), entity (<code>'s</code>), and
spaced forms.</li>
<li><span class="check">✓</span> <strong>ACME: replace the
self-signed bootstrap placeholder.</strong> When the secure server
starts with no managed certificate it writes a temporary self-signed
certificate so port 443 can bind, and with Auto SSL Certs on the
renewal job is supposed to replace it with a real one. It never did:
the placeholder covers the configured domains and is valid for a
year, so the renewal check (which renews only when the domains no
longer match, an authority renewal window opens, or the certificate
is within a month of expiry) saw nothing to do and left the
placeholder in place forever. The renewal check now recognizes the
placeholder — a self-signed certificate names itself as its own
issuer, so its issuer and subject match, unlike a certificate from an
authority — and treats it as due for renewal, so a fresh
install moves from the placeholder to a real certificate on the first
check. Once the real certificate is installed the normal
window/expiry logic takes over, and a failed obtain retries within
the hour.</li>
<li><span class="check">✓</span> <strong>Recommendation job:
pad short stored term embeddings.</strong> The job was logging a
stream of "Undefined array key" warnings while building wiki term
embeddings. Each term's embedding is a fixed-length vector of
doubles; the job reads one, adds weighted contributions at
hash-chosen positions, and writes it back. A vector stored before
the configured embedding size was increased (or one that failed to
decode) unpacks to fewer positions than the current size, so adding
a contribution at a high position touched a position that did not
exist yet and warned. The reader now pads any stored vector with
zeros up to the current size before returning it, so every caller
gets a full-length vector; short vectors are healed to the current
size the next time they are written.</li>
<li><span class="check">✓</span> <strong>getPages: don't freeze
the cooperative server during a fetch, and fix a curl busy-loop.</strong>
The multi-page fetcher waits for transfers in a loop that, inside the
cooperative web server, blocked the single event loop for the whole
fetch — so while the name server was fetching from other
machines it could not answer any other request, including a media
updater asking for its job list, which is how ACME (a media job)
could never get a turn to run. When the fetch loop runs inside a
request fiber it now hands the loop back between rounds of work so
other connections keep being served (restoring the error handler
around the hand-off so the work that runs in its place sees the
normal one); outside the server, on the crawler or a stand-alone
media updater, behaviour is unchanged except for one fix. That fix:
the wait call returns -1 when the underlying library has no socket to
wait on yet (for example mid-DNS), returning immediately, so the loop
could spin at full CPU; a short sleep on that case keeps it calm.
This is the missing half of getting ACME to run on a multi-machine
instance — the placeholder-replacement check from earlier only
helps once the job actually runs.</li>
<li><span class="check">✓</span> <strong>Network query: tolerate
a non-numeric timing field from a machine.</strong> Once the media
updater could run jobs again, the trending-highlights job ran a
multi-machine query and hit a fatal: a machine's serialized reply
carried an empty (or otherwise non-numeric) elapsed/total time, and
number_format rejects a non-numeric string outright, taking down the
whole job. The existing guard only filled in a missing field, not a
field present but non-numeric. The two timing values from each
machine are now coerced to numbers before formatting, so one
machine's malformed reply no longer crashes the query; a real reply
formats exactly as before.</li>
<li><span class="check">✓</span> <strong>Auto-derive the TLS
server context from the managed certificate.</strong> System Settings
reported "No certificate configured in SERVER_CONTEXT; secure mail
ports cannot start" whenever the operator left SERVER_CONTEXT out of
LocalConfig.php, even with Auto SSL Certs on and a real certificate
sitting on disk. The secure web server already binds straight from
the managed certificate and key files, but the mail side (and the
settings check) only looked at SERVER_CONTEXT, so the certificate
Auto SSL had obtained was invisible to them. Yioop now fills in
SERVER_CONTEXT from those certificate and key files automatically
when the operator has not pinned one and the files exist, so the
secure mail ports and the settings panel find the certificate with
no extra config line. An explicit SERVER_CONTEXT in LocalConfig.php
still takes precedence.</li>
<li><span class="check">✓</span> <strong>Let the site send its
own mail through its own mail server as the bot, with no stored
password.</strong> The site's outgoing mail (registration mail and
bulk mail) can now go out through the local mail server using the
reserved bot account, authenticated by a short-lived shared secret
rather than a password an operator has to set and store. When the
SMTP client logs in as <code>bot</code> it generates a fresh random
secret, writes it to <code>work_directory/security/bot_security.txt</code>
(owner-readable only, beside the TLS key), and sends it as the
password; the mail server's authenticator, seeing the bot login,
compares the supplied password against that file in constant time
instead of a stored credential, and on a match empties the file so
the secret is single use and cannot be replayed (a wrong guess
leaves it in place so it cannot wipe a secret a real send is about
to use). Because the sender and the mail server are the same
machine, the file is always shared between the two ends that need
it. A unit test covers the accept-and-consume, reject, replay,
empty-file, missing-file, and bot-name cases. Along the way: the
SMTP client field on the bulk-mail job was renamed from
<code>mail_server</code> to <code>smtp_client</code> and the same
variable in the registration controller from <code>server</code> to
<code>smtp_client</code> (both hold an SmtpClient, not a server),
and a pointless local copy of the username in the password check was
removed.</li>
<li><span class="check">✓</span> <strong>Move Account
Registration below Mail Services on Server Settings.</strong> The
Account Registration fieldset now sits after Mail Services, so the
mail-mode choice that will drive the upcoming "handled by MailSite"
registration option is made first. This is a pure reorder of the
fieldset; its contents are unchanged.</li>
<li><span class="check">✓</span> <strong>Derive the outgoing
mail connection when the bot route is on.</strong> A new setting,
MAIL_BOT_HANDLED, says the site's own mail (registration and bulk
mail) is sent through the built-in MailSite as the bot rather than
an external relay. Both send paths now build their mail client from
one factory, MailSiteFactory::outboundSmtpClient, instead of each
constructing one from the raw mail-server settings. When the bot
route is on the factory derives the connection: it connects to the
local MailSite on the submission port as the reserved bot account
over STARTTLS, sends from bot@<first-domain>, accepts the
local certificate without name checking (it is the same machine),
and carries no stored password since the bot secret is minted at
login. Otherwise it builds the client from the configured external
relay fields exactly as before. A unit test covers both routes. The
user interface that turns the setting on is the next step.</li>
<li><span class="check">✓</span> <strong>Account Registration
user interface for the MailSite route.</strong> The Account
Registration fieldset now offers a "MailSite Handled" checkbox and
relabels "Send Mail From Media Updater" to "Use a Queue". When Mail
Services is set to run MailSite (mailsite or both), switching
registration to Email Activation shows just the two checked boxes,
MailSite Handled and Use a Queue, and the manual mail-server fields
are hidden because their values are derived. Unchecking MailSite
Handled reveals those fields again. When MailSite is not in use the
checkbox is disabled and unchecked, so the manual fields are always
shown. The checkbox saves to a new MAIL_BOT_HANDLED profile field,
which is what the send path reads to choose the derived bot route
over the manual relay. This completes the feature: an operator can
turn on bot+MailSite sending entirely from Server Settings with no
config-file editing.</li>
<li><span class="check">✓</span> <strong>Treat admin activation
like email activation in the registration UI.</strong> Admin
activation also sends mail (a notice to the admin that an account is
waiting), so switching to it now defaults the MailSite Handled and
Use a Queue boxes the same way email activation does, rather than
only doing so for email activation. Any registration type that sends
mail gets the same default.</li>
<li><span class="check">✓</span> <strong>Make MAIL_SENDER the
site's single bot identity, readable by root.</strong> Instead of a
hardcoded "bot" name, the configurable MAIL_SENDER field (default
"bot") is now the bot identity everywhere the built-in MailSite is
used: it is the login, envelope sender, and From address when the
site sends its own mail, the address the List-Unsubscribe header
points at, and the mailbox name. MailSiteFactory gained botLocalPart
and botAddress helpers that read MAIL_SENDER (a bare name gets the
first local domain appended; a full address is used as is), and the
login check, outbound client, and unsubscribe address all key off
them. So the admin can read the bot's mail, saving Server Settings
with MailSite in use makes the bot address an alias of the root
account; mail to the bot is then delivered into root's INBOX. The
unsubscribe job follows: it now scans root's INBOX and acts only on
messages whose subject begins "unsubscribe " and that were delivered
to the bot address, removing just those and leaving root's other
mail alone. In the Account Registration form MAIL_SENDER stays
visible even under MailSite Handled (it is the bot address the
operator may want to set), while the rest of the manual mail-server
fields remain hidden. Also fixed the spacing between the Mail
Services and Account Registration fieldsets, which shared a margin
wrapper after the earlier reorder, so it now matches the other
fieldset gaps. Note: MAIL_SENDER's local part is the bot identity,
so it should not be the name of a real account.</li>
<li><span class="check">✓</span> <strong>Disable MailSite
Handled when MailSite is not in use.</strong> The MailSite Handled
checkbox is now rendered unchecked and greyed out from the server
side whenever Mail Services is not set to run MailSite (anything
other than the mailsite or both modes), rather than relying only on
a script to do it after the page loads. So when the site sends only
through external mail accounts, the checkbox cannot be left looking
checked.</li>
<li><span class="check">✓</span> <strong>Sender Email default,
validation, and local-part guard.</strong> The Sender Email field
now shows a full address: an empty or bare-name setting appears as
the bot address on the first mail domain (the default
bot@<primary-domain>) instead of blank or a bare name. When the
site runs its own mail, saving Server Settings now checks the sender:
if it carries a domain, that domain must be one of the site's mail
domains, and its local part must not already be the name of an
account (so the bot identity cannot shadow a real user's mail login).
Either problem stops the save with a clear message rather than
storing a sender the bot route could not use.</li>
<li><span class="check">✓</span> <strong>Update the bot-mailbox
tests for the root-INBOX behavior.</strong> The bulk-mail tests still
described the old dedicated bot mailbox that cleared every message.
They now match the current behavior: an unsubscribe message
addressed to the bot is suppressed and removed, while a message that
is not an unsubscribe to the bot (a stray message, or an unsubscribe
addressed elsewhere) is left in root's INBOX untouched.</li>
<li><span class="check">✓</span> <strong>Registration UI reacts
live to the Mail Services choice.</strong> Switching Mail Services to
External Mail Accounts (without saving) used to leave the MailSite
Handled checkbox checked and enabled, because the script read whether
MailSite was in use from a value baked in at page load rather than
from the current dropdown. The check now reads the live Mail Services
value, and changing Mail Services re-runs the registration sync, so
selecting External Mail Accounts immediately greys out and clears
MailSite Handled and brings the manual relay fields back. The sync
itself became a single flat helper in the style of
selectMonetizationType, leaning on setDisplay rather than hand-rolled
branches.</li>
<li><span class="check">✓</span> <strong>Server-Settings show
and hide scripts simplified to the shared helpers.</strong> The
page's own mail scripts were hand-rolling element lookups and direct
style.display writes where the shared basic.js helpers already do
the job. showHideWebServer and the insecure-delivery toggle now call
setDisplay (which also keeps aria-hidden in step), the insecure
toggle binds its change handler with listen, and the Suggested DNS
Records link calls toggleDisplay directly, so its one-line wrapper
function is gone. The dynamically built per-domain routing panels
keep their direct style writes because those elements have no id for
the id-based helpers to find.</li>
<li><span class="check">✓</span> <strong>Login no longer pads
every sign-in out to a fixed interval.</strong> Apple Mail's
Connection Doctor showed every AUTH on pollett.org taking about a
second. The cause was a "crude avoid timing attacks" block
in <code>SigninModel::checkValidSignin</code> that slept each login
up to 0.25 / 0.5 / 1.0s of wall clock. That pad was both slow and
redundant: the routine already runs a real bcrypt in the
missing-user branch, so the expensive work is already constant
whether or not the account exists. Removed the sleep and changed the
password compare from <code>==</code> to <code>hash_equals</code>, so
the timing-attack defence is now the constant bcrypt plus a
constant-time compare instead of a wall-clock pad.</li>
<li><span class="check">✓</span> <strong>Password hashing runs
inline; the per-login helper process is gone.</strong>
<code>crawlCrypt</code> handed each bcrypt to a freshly spawned
<code>php -r</code> helper so the cooperative loop would not block.
A standalone probe put cost-12 bcrypt at ~0.29s on the (x86_64)
pollett.org box — less than the ~0.09s it costs just to spawn
the helper, plus the Apple-Silicon scheduling variance the offload
added (one run swung an AUTH to 1.3s). On a single-process personal
server logins rarely overlap, so the offload was paying that
overhead on every login for a benefit never collected. Dropped the
offload (<code>crawlCryptOffload</code> removed) and compute
<code>crypt()</code> inline: deterministic ~0.3s, no spawn. The
bcrypt cost stays at 12 (the PHP 8.4 / Laravel 11 default). A
separate experiment that turned the daemon's parked-fiber busy-spin
into a 2ms sleep was tried and reverted — on Apple Silicon the
sleeping loop let the hash helper drift onto efficiency cores and
ran slower, so the spin stays.</li>
<li><span class="check">✓</span> <strong>Standard-folder
provisioning lists the mailbox once, not once per role.</strong>
With the hash no longer dominating, in-daemon instrumentation showed
the rest of an IMAP login was <code>ensureStandardFolders</code>,
run on every login, at ~355ms (and several seconds on a cold cache).
It called <code>listFolders</code> inside its per-role loop, walking
the whole mailbox tree four times even when every canonical folder
already existed and nothing needed consolidating. It now lists once
and reuses the result, creating a canonical folder only when it is
missing and entering the message-move path only when an alias folder
("Spam", "Sent Items") is actually present.</li>
<li><span class="check">✓</span> <strong>Folder listing is
cached per user and stays correct across processes.</strong> Added a
per-user in-memory folder list, validated against a small per-user
<code>folder_generation.txt</code> token that
<code>createFolder</code>, <code>deleteFolder</code>, and
<code>renameFolder</code> rewrite (staged to a temp name and renamed
into place) on any change in any process.
<code>listFolders</code> reads the token first and only re-walks
when it differs, so a folder created or deleted in the web interface
shows up live in IMAP, while an unchanged mailbox costs one small
read instead of a tree walk. <code>collectFolderTree</code> also
rejects <code>.eml</code> message files with a single suffix test
before its fuller metadata checks. Net for this run: IMAP AUTH on
pollett.org fell from ~1.08s to ~0.35–0.44s, now floored by
the cost-12 hash itself.</li>
<li><span class="check">✓</span> <strong>BASE_URL became a
function so mail (and every) link names the domain in use, not
localhost.</strong> Account-activation and password-recovery mail
from <code>yioop.com</code> carried links reading
<code>https://localhost/</code>. <code>BASE_URL</code> was a
constant built once at start-up from
<code>$_SERVER['SERVER_NAME']</code>; under the single-process atto
server that is the one name the server bound to (often localhost)
and never the domain a given visitor used, and a server answering to
several domains has no single correct constant. Since the value is
genuinely per request, it can no longer be a constant: it is now a
function <code>C\baseUrl()</code> in <code>Config.php</code>,
computed nowhere else. It reads the host (and scheme and port) from
<code>$_SERVER</code>, which the web server in front of Yioop sets
per request — atto, Apache, and nginx alike — so the
same code is correct under any of them; the fixed path part stays
the <code>SHORT_BASE_URL</code> constant. The bootstrap that used to
assemble the constant was reduced to setting that path and seeding
<code>SERVER_NAME</code> from the launch address; the scheme rule
(Apache's <code>HTTPS</code> "off", and the
own-web-server TLS-context fallback gated on
<code>IS_OWN_WEB_SERVER</code>) moved into the function. Every
<code>C\BASE_URL</code> use across the tree (controllers, models,
views, library) was changed to the call; the three
<code>nsdefined("BASE_URL")</code> guards collapsed into
the function, which returns <code>NAME_SERVER</code> for the
command-line tools that previously relied on that fallback. Outside
a request, where no host is in <code>$_SERVER</code>, the host falls
back to the first configured served name and then the
operating-system name from <code>gethostname</code>. Under atto the
served name is made trustworthy at a lower level than the
controller: the launcher passes the parsed
<code>SECURE_DOMAINS</code> into the <code>WebSite</code> config
(top-level, so it survives secure mode where
<code>SERVER_CONTEXT</code> is unset), and the self-contained
<code>WebSite</code> sets <code>$_SERVER['SERVER_NAME']</code> per
request to the visitor's <code>Host</code> when it is one of those
names, or the first served name otherwise — the allowlist
check keeps the attacker-controllable <code>Host</code> header out
of the links.</li>
<li><span class="check">✓</span> <strong>Test suite made green
and re-run safe.</strong> Three test fixes. First,
<code>UtilityTest</code>'s cooperative-hash case still asserted the
old behaviour — that inside a fiber <code>crawlCrypt</code>
offloads to a helper and suspends on its pipe — which the
inline-hashing change had removed; it now asserts the real
behaviour, that the call runs to completion in one step inside a
fiber and returns the same hash as a direct call. Second, an
intermittent failure that surfaced only after the suite had been
run many times in a row on one work directory was root-caused to
the data-source connection cache. The mail-model tests
(<code>MailCloneModelTest</code>, <code>MailAccountModelTest</code>,
<code>MailSuppressionModelTest</code>,
<code>MailAliasModelTest</code>) each open a throwaway database
whose file name carries only a small random tag, and the data-source
layer keeps a process-wide cache of open handles keyed by path. A
plain disconnect nulls the handle but does not drop it from that
cache, so when two tests in one run happened to draw the same tag
the second was handed the first's still-open handle — rows and
all, since the unlinked file stays alive through the handle —
and assertions such as "two claimable jobs" saw leftovers
and failed. Each of these tests now evicts its own handle from the
cache in <code>tearDown</code> (keyed by the connection string, so
shared handles are left untouched). Third, the <code>UnitTest</code>
runner called a test method and then <code>tearDown</code> without a
finally, so a method that threw part way through skipped its
<code>tearDown</code> and the eviction above; <code>tearDown</code>
now runs in a finally so it always cleans up.</li>
<li><strong>Keep member accounts and the bot off each other's
address (pending commit).</strong> The site's bot is the sender of
all system mail and owns the mailbox the bulk-mail job scans, so an
account and the bot must never share an address. Two directions are
now guarded. Account side: when an account is created or its email is
changed (self-registration, an admin adding or editing a user, and a
member editing their own profile), an email that is the bot's address
on a managed mail domain is refused;
<code>MailSiteFactory::isForbiddenAccountEmail</code> holds the rule,
with the decision in the testable <code>accountEmailRefused</code>.
The off-by-default <code>MAIL_REFUSE_LOCAL_DOMAIN_ACCOUNTS</code>
extends that to every address on a managed domain, so an in-house
install where everyone shares the domain still works by default. Bot
side: the existing check that refused renaming the bot onto an
existing <em>username</em> now also refuses renaming it onto an
address an existing account already uses as its <em>email</em> (new
<code>UserModel::getUserByEmail</code>). Covered by
<code>MailSiteFactoryTest</code> and <code>UserModelTest</code>. Also
reworded the login-code verify screen (title “Sign-in”,
field label “Emailed code”).</li>
<li><span class="check">✓</span> <strong>Standardize sign-in
wording, drop “login”, collapse duplicate labels.</strong>
The sign-in view
title is now always the single <code>signin_view_signin</code> =
“Sign in”; the password button, the emailed-code request and
verify screen titles, and the verify button all resolve to it, so four
duplicate labels (<code>signin_view_login</code> and the emailed-code
<code>_title</code>/<code>_enter_title</code>/<code>_verify_button</code>)
are removed. The welcome-menu and page-options links also read
“Sign in”; adjective uses such as “sign-in code”
keep the hyphen. The word “login” is removed from the
user-facing message texts (cookie and sign-in prompts, “Signed
in!!”). The emailed-code locale keys are renamed from
<code>*_login_code_*</code> to <code>*_email_code_*</code> with all
<code>tl()</code> references updated, so the keys no longer carry
“login”. The only emailed-code-specific labels left are the
field label and the “Email me a code” button.</li>
<li><span class="check">✓</span> <strong>Rename internal
<code>LOGIN_CODE</code> symbols to
<code>SIGNIN_CODE</code>; restore the emailed-code link
text.</strong> With the user-facing wording settled, the feature's
internal names now match: the <code>LOGIN_CODE</code> table becomes
<code>SIGNIN_CODE</code>, the <code>LOGIN_CODE_*</code> timing constants
become <code>SIGNIN_CODE_*</code>, and the controller and model members
(<code>signinCode</code>, <code>processSigninData</code>,
<code>signinCodeComplete</code>, <code>setSigninCode</code> and friends,
the <code>SIGNIN_SIGN_PREFIX</code> token tag, and the session and data
keys) drop “login” too. Because the table only ever existed
on the dev machine, the version-113 migration and the fresh schema are
edited in place to create <code>SIGNIN_CODE</code> directly — no
new migration, <code>DATABASE_VERSION</code> stays 113. The emailed-code
entry link, which the wording pass had over-collapsed into plain
“Sign in”, is restored to “Sign in with emailed
code”. <code>SigninModelTest</code> updated and green.</li>
<li><span class="check">✓</span> <strong>Replace the fixed recovery
quiz with a member-typed recovery question, for airgapped sites (item
33c).</strong> The old “email link and check questions”
mode generated a like/dislike quiz over six fixed, translated questions
and stored the answers in the session. It is replaced by a single
recovery mode, <code>QUESTIONS_RECOVERY</code>, in which each member
types their own question and answer. Two columns are added to the
<code>USERS</code> table — <code>RECOVERY_QUESTION</code> (kept as
plain text so it can be shown back) and <code>RECOVERY_ANSWER</code>
(only a one-way hash is stored, the same way a password is) — via
a new <code>upgradeDatabaseVersion114</code> migration, so
<code>DATABASE_VERSION</code> is now 114. <code>UserModel</code> gains
<code>setRecoveryQuestion</code> and a constant-time
<code>checkRecoveryAnswer</code>. Registration, the Manage-Account
security panel, and the airgapped recover screen now show a plain
question field and an answer field instead of the quiz dropdowns; the
recover screen is a single page that, once the username and captcha
clear, shows the member their own question with an answer box and the
new-password fields together. Email-link recovery is unchanged. After
sign-in, a member with no question set yet is reminded to add one. The
quiz machinery is removed end to end (the
<code>getRecoveryQuestions</code>,
<code>selectQuestionsAnswers</code>, <code>checkRecoveryQuestions</code>
and <code>makeRecoveryQuestions</code> methods, the
<code>NUM_RECOVERY_QUESTIONS</code> constant, the
<code>register_view_recovery</code> question strings, and the
session-based answer storage), and the obsolete
“edit recovery questions” locale link is dropped.
<code>UserModelTest</code> covers the store/verify round trip and the
empty-answer case.</li>
<li><strong>Fix the Change-Recovery-Question save and finish the
old-quiz removal (pending commit).</strong> The
Manage-Account “Change Recovery Question” form never
persisted: the save branch still guarded on the deleted
<code>$recovery_answer_change</code> flag and saved a session blob
rather than calling <code>setRecoveryQuestion</code>, so a typed
question and answer were silently dropped — which then made
question-based password reset fail because no answer had been stored.
The save branch now stores the question and answer (the answer hashed,
the same as at registration) once the confirming password checks out.
Recovery answers are verified against the raw typed text so storage and
verification hash the same value. The last dead remnants of the old
quiz are removed: the <code>$recovery_answer_change</code> guard and the
now-unused <code>RegisterController</code>, <code>LocaleModel</code> and
<code>RegisterView</code> imports the deleted
<code>makeRecoveryQuestions</code> had pulled in.</li>
<li><strong>Show that a recovery answer is already on file (pending
commit).</strong> On the Manage-Account “Change Recovery
Question” form the answer box was always blank, giving no sign an
answer had been saved. It now shows a fixed run of bullets when an
answer is on file. The answer itself stays a one-way hash and is never
shown or stored in the clear; the bullets are only a marker. Saving with
the bullets left untouched rewords the question and keeps the existing
answer (<code>setRecoveryQuestion</code> now takes a null answer to mean
“leave it”); typing over the bullets sets a new answer. The
marker lives in one place as <code>RECOVERY_ANSWER_PLACEHOLDER</code>.
<code>UserModelTest</code> covers the question-only edit keeping the old
answer.</li>
<li><span class="check">✓</span> <strong>HTTP/2 partial transfer
over a real network.</strong> The wiki Media-List edit page (about
185 KB) intermittently failed in Firefox with
<code>NS_ERROR_NET_PARTIAL_TRANSFER</code>: it downloaded about
128 KB, hung for roughly a minute, then finished or gave up;
Safari and Chrome were fine and loopback never reproduced it. A
Firefox log capture pinned it: Firefox enlarges a stream's window to
about 12 MB with a WINDOW_UPDATE the instant the stream opens,
but the server was not reading it. Over TLS, OpenSSL decrypts a whole
record into its own buffer while <code>stream_select</code> watches
only the raw socket, so a single read per wakeup took the request
headers and left the grant sitting decrypted-but-unseen until an
unrelated wakeup (Firefox's 58-second keep-alive) finally read it.
Three fixes: the read path now drains the decrypted buffer to empty
each wakeup, capped at the chunk size and stopping on the first empty
read, so the grant is read alongside the headers and the stream never
parks; the end-of-file close now waits until nothing is left to send
(no queued bytes, no parked send, no streaming generator), since the
drain exposes a spurious end-of-file on a non-blocking TLS stream that
used to tear the connection down mid-response; and the frame loop now
dispatches every request that arrived in one read, not just the first,
so a page plus its style sheets, scripts, and images all load instead
of stalling at zero bytes. The investigation tracing and an earlier
ping-on-park nudge were then removed. <code>H3Listener</code> was
reviewed and needs none of these: QUIC datagrams are discrete, its
read loop already drains every datagram, its drive step already
iterates every stream, and QUIC's own close replaces TCP end-of-file.
Covered by <code>readCreditThenKeepsFlushingPastEofTestCase</code> and
<code>readDispatchesEveryRequestInOneReadTestCase</code>.</li>
<li><span class="check">✓</span> <strong>HTTP/2 closed-stream
window updates no longer exhaust the credit guard.</strong> The drain
above had a second consequence that only showed on a page streaming
many ranges in a row, such as a playing video the browser fetches a
chunk at a time: each range is its own short-lived stream, and a
browser sends WINDOW_UPDATE frames as it consumes a response, so some
arrive just after the server has finished a range and closed that
stream. Before the drain those late frames were usually left unread;
with the drain they are all read. The handler could not tell a
just-closed stream from a freshly-opened one awaiting its response
— both have no send window yet — so it stashed the late
credit as a pre-grant that nothing ever consumed, accumulating one
stranded entry per closed stream until it reached the
max-concurrent-streams ceiling and the next grant was misread as a
misbehaving peer and answered with a GOAWAY, surfacing as a 206 that
returned no bytes and a video that stopped after about a hundred
chunks. The server now remembers the most recently closed streams
(bounded, oldest dropped) and ignores a WINDOW_UPDATE for one of them,
which RFC 7540 sec 6.9 permits, while still holding a genuine
pre-grant for an open stream so the larger-than-window page path above
is untouched. Covered by
<code>closedStreamWindowUpdateForClosedStreamIgnoredTestCase</code>
and <code>preGrantForOpenStreamStillAppliedTestCase</code>.
<br><em>A close-races-the-tail case on a one-shot connection (a
connection the client closes immediately after a single
larger-than-window response) is older than this work and still open;
a connection kept open across requests, the browser case, never lost
bytes in testing.</em></li>
</ol>
</body>
</html>
(c) 2026 Seekquarry - Open Source Search Engine Software