<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Page Source / History Visibility and Robots — Arc Plan</title>
<style>
body { font-family: system-ui, sans-serif; max-width: 760px;
margin: 1.5em auto; padding: 0 1em; line-height: 1.5;
color: #1a1a1a; }
h1 { font-size: 1.4em; }
h2 { font-size: 1.1em; margin-top: 1.4em; }
ol { margin: 0.3em 0; }
li { margin: 0.25em 0; }
.check { color: #1a7f37; font-weight: bold; }
.next { color: #b35900; font-weight: bold; }
.q { color: #8250df; }
code { background: #f2f2f2; padding: 0 0.25em; border-radius: 3px; }
em { color: #555; }
</style>
</head>
<body>
<h1>Page Source / History Visibility and Robots</h1>
<p style="margin:.8rem 2rem;padding:.7rem 1rem;background:#eef2f7">
<strong>Arc Post-Mortem Summary.</strong>
Three parts covering wiki page visibility and
site robots files. Part 1 fixed source and history visibility rules for wiki
pages; Part 2 handled public page-list visibility; and Part 3 built
per-domain robots.txt and humans.txt, covering their generation, serving, and
the edit-page wiring.</p>
<p><em>Arc started 2026-06-16. Goal: let a group owner keep the
public (and crawlers) out of wiki page source and history, and let
each web-server domain publish its own robots.txt / humans.txt.
Group-level CSS was considered and dropped — domain-level CSS
already covers that need.</em></p>
<h2>Part 1 — Source / history visibility
<span class="check">✓ done (this patch)</span></h2>
<ol>
<li><span class="check">✓</span> <b>OPTIONS becomes a flag
bitfield; render engine gets its own column.</b> Previously
<code>OPTIONS = (render_engine << 1) + encrypted</code>,
which left no spare bit. Now:
<ol>
<li><span class="check">✓</span> New column
<code>SOCIAL_GROUPS.RENDER_ENGINE</code> (fresh schema in
<code>ProfileModel</code>).</li>
<li><span class="check">✓</span> <code>OPTIONS</code> holds
only yes/no flags: bit 0
<code>GROUP_OPTION_ENCRYPTED</code>, bit 1
<code>GROUP_OPTION_PAGE_SOURCE_ALLOWED</code> (new constants in
<code>Config.php</code>).</li>
<li><span class="check">✓</span> <code>GroupModel</code>
pack/unpack updated: <code>addGroup</code>,
<code>updateGroup</code>, <code>isGroupEncrypted</code>,
<code>getRenderEngine</code> (reads the column),
<code>getGroupById</code> (selects the column, unpacks both
flag bits). Public method signatures unchanged, so callers do
not move.</li>
<li><span class="check">✓</span> Migration
<code>upgradeDatabaseVersion106</code>
(<code>DATABASE_VERSION</code> 105→106): adds the column,
then in PHP (portable across all five PDO back ends) copies each
group's render engine out of OPTIONS and rewrites OPTIONS to
keep the encrypted bit and turn the page-source-allowed bit on,
so existing behaviour is preserved.</li>
</ol></li>
<li><span class="check">✓</span> <b>Group-level master.</b>
<code>PAGE_SOURCE_ALLOWED</code> (the bit) controls public view of
both source and history. Default allowed. Set from a new "Public
Source & History" Allowed / Not Allowed dropdown in the group
edit form (<code>ManagegroupsElement</code>), wired through
<code>PAGE_SOURCE_CODES</code> and the
<code>updateGroup</code> field list.</li>
<li><span class="check">✓</span> <b>Page-level override.</b>
The existing per-page <code>public_source</code> head var now gates
history as well as source. Either the group flag or the page var
being off denies the public; editors and owners always see both.</li>
<li><span class="check">✓</span> <b>Enforcement.</b> The
history endpoint now returns 404 to the public user when the group
disallows, or when the current page's <code>public_source</code> is
off. <code>NO_HISTORY_SOURCE</code> also fires on the group flag, so
the Source and History links disappear from the page for the
public.</li>
<li><span class="check">✓</span> <b>Page settings cue.</b> When
the group disallows, the page-settings <code>public_source</code>
checkbox is disabled and shows an "(off at group level)" note, so an
editor is not misled into thinking a page can opt back in.</li>
<li><span class="check">✓</span> Four locale strings added to
<code>src/locale/en_US/configure.ini</code>.</li>
<li><span class="check">✓</span> <b>Revision (2026-06-17): hide
the buttons, not just the bodies.</b> Source, history and discussion
controls now disappear when the user may not view source, keyed off
the same <code>NO_HISTORY_SOURCE</code> signal (page var or group
flag off, owners exempt): the floated source/discuss buttons
(<code>GroupbarElement</code>), the page-mode menu's
source/history/discussion entries (<code>GroupmenuElement</code>),
and the page-listing "view source" button
(<code>WikiElement</code>, gated on the group flag).</li>
<li><span class="check">✓</span> <b>Revision (2026-06-17):
a page's discussion thread follows its source.</b> When source is
disallowed for the public user, the discussion thread itself
returns 404, not just its hidden link — enforced where the
feed path already resolves a thread back to its wiki page via
<code>getPageInfoByThread</code>.</li>
</ol>
<p><em>Arc folders for a shared day are now suffixed in the order they
happened (2026-06-16A/B/C); existing 2026-06-16 arcs were relabelled
accordingly. Past Z, suffixes double (AA, AB, …).</em></p>
<h2>Part 2 — public page-list visibility
<span class="check">✓ done (2026-06-17)</span></h2>
<ol>
<li><span class="check">✓</span> New OPTIONS flag bit
<code>GROUP_OPTION_PAGE_LIST_ALLOWED</code> (bit 2, value 4)
controlling whether the public may see a group's list of wiki pages.
Default allowed.</li>
<li><span class="check">✓</span> Migration
<code>upgradeDatabaseVersion107</code>
(<code>DATABASE_VERSION</code> 106→107): turns the bit on for
every existing group in PHP, so the listing stays visible as
before.</li>
<li><span class="check">✓</span> <code>GroupModel</code> packs
and unpacks the bit in <code>addGroup</code>, <code>updateGroup</code>
and <code>getGroupById</code> alongside the other flags.</li>
<li><span class="check">✓</span> Set from a new "Public Page
List" Allowed / Not Allowed dropdown in the group edit form
(<code>ManagegroupsElement</code>), wired through
<code>PAGE_LIST_CODES</code> and the field list.</li>
<li><span class="check">✓</span> Enforcement: the "pages"
listing endpoint returns 404 to the public user when the group
disallows it, and the "create / find pages" menu link is hidden
from the public (<code>GroupmenuElement</code>). Members and owners
are unaffected.</li>
<li><span class="check">✓</span> Two locale strings plus the
dropdown label added.</li>
<li><span class="check">✓</span> <b>Fix (2026-06-17):</b> the
create-group form reads <code>page_source_allowed</code> and
<code>page_list_allowed</code> from the default group array, so
both keys are seeded (allowed) there to stop undefined-key warnings
on the Create Group page.</li>
</ol>
<h2>Part 3 — per-domain robots.txt / humans.txt
<span class="next">▶ NEXT</span></h2>
<p><em>Design agreed 2026-06-17 from Chris's spec. Each secure-route
domain gets its own generated robots.txt and a free-form humans.txt,
edited from the Web Server settings. Decisions: store in the per-domain
appearance blob (no migration); a disallowed category blocks every
address that reaches it (both the clean path and the query form);
toggles are global for now (not per-group); the old shipped
src/robots.txt is removed since this supersedes it.</em></p>
<h2>Part 3a — generation & serving
<span class="check">✓ done (2026-06-17)</span></h2>
<ol>
<li><span class="check">✓</span> <code>RobotsTxtGenerator</code>
library class (pure logic): turns a domain's saved yes/no choices
into robots.txt text. Disallow-all yields a blanket
<code>Disallow: /</code>; otherwise it opens for all crawlers, adds
a crawl-delay line, a Disallow block for every address leading to a
turned-off category (both clean path like <code>/group</code> and
query form like <code>/*?*c=group</code>), then the admin's extra
directives. An unconfigured domain defaults to everything crawlable
except search-result addresses.</li>
<li><span class="check">✓</span> Serving: index.php route map
now sends <code>robots.txt</code> to <code>routeRobotsTxt</code> and
<code>humans.txt</code> to <code>routeHumansTxt</code>. Each finds
the request host, reads that domain's saved config from the
appearance blob via a <code>GroupModel</code>, and emits generated
text through the WebSite object, exactly as the
<code>mta-sts.txt</code> route does.</li>
<li><span class="check">✓</span> Removed the old static
<code>src/robots.txt</code>; the generated default supersedes it.</li>
</ol>
<h2>Part 3b — the edit page & wiring
<span class="check">✓ done (2026-06-17)</span></h2>
<ol>
<li><span class="check">✓</span> "Configure robots.txt and
humans.txt" link beneath the Routing row in the Web Server domain
panel, opening <code>serverSettings&arg=robots&domain=...</code>.
</li>
<li><span class="check">✓</span> The edit page itself: a new
element with a <code>CloseHelper</code> [x] floated opposite; the
disallow-all checkbox (JS-hides the other robots inputs, humans
stays); the crawl-delay dropdown (1/3/5/10/30/60); the
search/feeds/wiki checkboxes with the three wiki sub-checkboxes
(source & history, page feeds, page lists); the additional-robots
and humans.txt textareas; and Save / Cancel / Preview (Preview opens
a tab with the generated robots.txt).</li>
<li><span class="check">✓</span> <code>serverSettings</code>
handling: render the page for <code>arg=robots</code>, save the
choices into the domain's appearance blob (merging, so appearance
overrides are untouched), and a preview path that runs the same
generator.</li>
<li><span class="check">✓</span> Locale strings for the link, the
control labels, and the buttons.</li>
<li><span class="check">✓</span> Polish (2026-06-17): the config
link sits on its own line beneath Routing (full-width flex row); saving
no longer asks for a server restart (the files are served from the DB);
and Save / Cancel / Preview are all the same button element so they
match in size. <em class="q">Cancel is now a submit that returns to
Server Settings with no message.</em></li>
</ol>
<ol>
<li><span class="check">✓</span> Store each domain's robots.txt
and humans.txt text alongside <code>DOMAIN_APPEARANCE</code> (the
per-domain serialized override table from the v104 migration).</li>
<li><span class="check">✓</span> Add "configure robots.txt" /
"humans.txt" links to the web-server
domain edit panel (<code>ServersettingsElement</code> +
<code>SystemComponent</code> domain routes), each opening a simple
textarea editor.</li>
<li><span class="check">✓</span> Resolved (was an open question
about whether crawlers hitting group pages should be steered by this
robots.txt by default): the two are orthogonal. robots.txt only advises
crawlers which paths to fetch; the group-level page-source master from
Part 1 enforces what the public may view (server-side 403/404). One
does not steer the other, so there is no per-group robots default to
decide.</li>
</ol>
</body>
</html>
(c) 2026 Seekquarry - Open Source Search Engine Software