Implements the remediation pass described in
planning/19-security-audit-2026-04-30.md. All Critical findings and the
Wave 1-4 High findings are closed; PoC tests added; full backend test
suite green; helm chart lints clean.
Wave 1 - Auth & identity
- C1 OAuth state nonce: PutOAuthState / ConsumeOAuthState (valkey,
GETDEL single-use, 15min TTL); Callback rejects missing/forged/cross-
provider state before token exchange.
- C2 OAuth identity linking: refuse silent linking to existing user
unless info.EmailVerified is true. fetchGitHubUser now consults the
/user/emails endpoint for the verified flag (no more hardcoded true);
fetchFacebookUser sets EmailVerified=false (FB exposes no per-email
verification flag).
- H1 Magic-link verify: replaced Get + MarkUsed with a single atomic
UPDATE...RETURNING (ConsumeMagicLink) - TOCTOU-free.
- H2 TOTP code replay: MarkTOTPCodeConsumed (valkey SET NX, 120s TTL)
prevents replay of a successfully validated code; fails closed on
transient store errors.
- H3 Backup-code orphan: DisableTOTP now also wipes totp_backup_codes.
Wave 2 - Middleware & network
- C3 CORS/CSRF regex anchoring: NewCORSConfig wraps each pattern with
\A...\z so substring spoofing of origins is impossible.
- H4 ClientIP: server reads APP_TRUSTED_PROXIES; gin SetTrustedProxies
is called explicitly (empty default = no proxy trust).
- H11 Body limit + DisallowUnknownFields: BodyLimitBytes middleware
(1 MiB default) wraps every request; validate.BindJSON now uses a
json.Decoder with DisallowUnknownFields and rejects trailing tokens;
413 envelope on body-limit overflow.
- H16 NetworkPolicy: backend.networkPolicy.enabled defaults to true;
new web-networkpolicy.yaml restricts web pod ingress to nginx-gateway
and egress to backend service + DNS + 443.
Wave 3 - Encryption at rest
- C4 TOTP secrets: CreateTOTPSecret writes encrypted secret_v2;
GetTOTPSecret prefers v2 with legacy fallback.
- C5 OAuth tokens: migration 000033 adds *_v2 columns; CreateOAuthAccount
and UpdateOAuthTokens write encrypted; GetOAuthAccount reads v2 with
legacy fallback.
- M1 Domain separation: crypto.DeriveKeyFor(secret, purpose) replaces
single-purpose DeriveKey; settings, totp, oauth each use a distinct
HKDF-derived subkey. DeriveKey kept as back-compat alias for settings.
Wave 4 - Input & AI safety
- C6 SSRF: new pkg/safehttp refuses to dial RFC1918, loopback, link-
local, ULA, multicast, unspecified, or cloud-metadata IPs; scheme
allowlist (http/https). Wired into pkg/scrape, discovery LinkChecker,
and imageURLReachable. NewForTesting opt-in for httptest.
- H13 PromptGuard German + Unicode: NFKC + Cf-class strip pre-pass
closes zero-width and full-width-homoglyph bypasses; new German rules
for ignoriere/missachte/vergiss/role-escalation/prompt-exfil/verbatim;
Gemma-style and pipe-delimited chat-template tokens covered;
source-fence rule prevents '=== Quelle:' splice in scraped text.
- H14 BudgetGate: new ai.BudgetGate interface; UsageRepo.CheckBudget
reads today's SUM(estimated_cost_usd) (10s cache) and refuses calls
when AI_DAILY_CAP_USD is exceeded; GeminiProvider.Chat checks the
gate before contacting Gemini.
OAuth routes remain disabled in server/routes.go, so C1/C2 are not
actively reachable today; fixes ensure correctness when re-enabled.
RevokeSession, RevokeSessionsByFamilyID, DeleteUserSessions,
RevokeOtherSessions, and ConsumeRefreshToken updated revoked_at in
Postgres but did not invalidate the valkey access-token cache. The cache
serves the original Session JSON (RevokedAt: null) until its TTL expires
(JWT_ACCESS_TTL = 2h), so logout / admin-revoke / refresh-reuse-detection
took up to 2h to actually invalidate.
Fix: each revocation path now uses RETURNING access_token_hash and DELs
the cache key via new helper invalidateCachedSessions. revokeBulk handles
multi-row revocations.
Adds three router-level negative tests for the admin auth chain
(RequireAuth + RequireRole("admin")):
- TestAdminChain_UserRole_Returns403 — user role rejected with 403
- TestAdminChain_AdminRole_Passes — admin role accepted
- TestAdminChain_NoBearerToken_Returns401 — missing token rejected with
401 (auth runs before role check)
Repository-level regression test for the cache invalidation requires
real Valkey + Postgres, currently not in test harness — flagged as TODO
in planning/18-security-threat-model.md.
Audit findings H1, E (negative tests for session validation, authz).
New pkg/promptguard.Sanitize strips known structural injection patterns
(role labels, override directives, chat-template tokens, llama tokens,
prompt-exfil) from third-party scraped content before it reaches Gemini.
Wired into both LLM call sites:
- discovery/enrich.ProviderLLMEnricher.EnrichMissing (per-source quellen)
- market/research.buildUserPrompt (quellePage title + text)
Defense-in-depth on top of existing structural framing (JSON envelope in
research, JSON-Schema constrained decoding in enrich_b).
Audit finding H2.
Two follow-ups to the apply-shape fix:
opening_hours: previous converter only kept datum_von's weekday, dropping
the rest of multi-day ranges. A Sat-Sun event ended up with only Samstag
saved. Now iterates [datum_von, datum_bis] inclusive, projects each date
to its German weekday, and dedupes by (weekday, open, close). Different
hours on the same weekday still produce separate rows so the admin sees
the conflict. Capped at 7 distinct weekdays — the form's expressive limit.
website validator: the LLM was emitting aggregator/listing URLs (e.g.
suendenfrei.tv detail pages) as the official event website because that's
where it found grounding info. The prompt already forbade this, but the
model ignores soft rules. Add a hard validator-level rejection for known
aggregator domains: suendenfrei.tv, mittelalterkalender.info,
marktkalendarium.de, festival-alarm.com, mittelaltermarkt.online. These
suggestions now land in the "rejected" bucket with a clear reason.
image_url / logo_url are unaffected (those legitimately come from any host).
Prompt: enumerate the same aggregator domains explicitly under both the
"primary source" fallback list and the "website forbidden" rule, so the
model has a concrete blacklist instead of a category description.
Existing markets with already-saved aggregator websites need manual
clearing — the validator only kicks in on new applies.
Tests: 4 new opening-hours subtests (range expansion, weekday dedupe,
cross-entry dedupe, hours-conflict-kept-separate) and 4 validator subtests
(three aggregator domains rejected as website; aggregator host as image_url
still ok).
When apply moved server-side (9b30863), the client-side conversion from
dd9a5ae was lost. opening_hours and admission_info were stored raw in the
LLM's German shape ([{datum_von,...}], [{betrag,name,waehrung}]), which the
form's reactive bindings could not parse — fields appeared empty after
Uebernehmen + reload.
applyFieldMerge now routes both fields through dedicated converters:
- admission_info: smart name-prefix mapping (Erwachsen*->adult_cents,
Kind*/Child*->child_cents, Ermaessigt*/Schueler*/Senior*/Rentner*/Reduced
->reduced_cents). Unmapped ticket names append to notes so admins still
see the extracted info. Already-form-shape input (map with adult_cents)
passes through unchanged.
- opening_hours: each LLM entry's datum_von is parsed as ISO date and
converted to a German weekday name. Entries with unparseable datum_von
are dropped (better than writing rows the form's day select cannot bind).
Already-form-shape input ([{day,...}]) passes through unchanged.
Forward-only: markets where a previously broken apply already wrote
LLM-shape JSON need a re-apply (or manual edit) to render correctly.
Tests: 8 new TestApplyFieldMerge_* subtests covering smart mapping, notes
fallback, weekday derivation, malformed dates, and pass-through.
CI deploy steps now target helm/marktvogt with --reset-then-reuse-values,
preserving the other service's image tag across pipeline runs. Each pipeline
sets only its own X.image.tag.
App-level secrets (smtp/turnstile/discovery/ai/JWT/oauth) moved out of CI's
--set chain in the previous phase — now pre-created via
scripts/k8s-secrets-sync.sh from .env.helm. The chart's conditional secret
templates remain for backward-compat with the live release's stored values
but will be removed in a follow-up once those values are cleared.
Old per-service chart directories deleted; only the monolithic
helm/marktvogt/ remains.
MIGRATION.md updated with the actual procedure that worked, including the
several pitfalls hit during the live tenant-2 migration on 2026-04-28
(helm uninstall trap, SSA field-manager swap for CRDs, kyverno hostname
allowlist for new subdomains).
Nil slices in MergePlan.AutoApply/ReviewRequired/Rejected serialized to
JSON null, causing the admin research panel to crash with
"can't access property 'map', plan.review_required is null". Initialize
the buckets as empty slices so the wire contract is always an array.
Tightened the empty-buckets test to assert the JSON shape.
Re-prices every existing ai_usage row using the correct $/1M token rates
per model family. CASE clauses ordered specific-first (flash-lite before
flash) to mirror the longest-prefix-match in priceFor(). Aliases
(gemini-*-latest) resolve to the 2.5 family, the only one in production
during the affected window.
The grounding-fee component ($35/1k above 1500/day free tier) is not
recomputed: historical traffic shows zero grounded calls in the window,
so the bumper would be 0. Down is a no-op (irreversible by design — the
original miscalculated values are not preserved).
estimateCost ignored the model name and billed every Gemini call at
hardcoded flash-lite rates ($0.10 / $0.40 per 1M), under-counting Pro
calls by ~12-25x. Switch to priceFor(model) and prefer resp.ModelVersion
so aliases like gemini-pro-latest resolve to their concrete family.
Capture ThoughtsTokenCount as a separate ThinkingTokens column on
ai_usage (migration 000030) and bill it at the output rate.
Add a global thinking on/off toggle that mirrors the grounding pattern:
provider holds an in-memory cache (read at startup from settings.Store),
handler keeps it in sync, Chat() applies ThinkingConfig.ThinkingBudget=0
only when disabled. Default true preserves SDK behavior. Grounding+
thinking get/set helpers folded into shared getBool/setBool to keep
goconst happy.
Web admin settings: new "Modell-Reasoning" toggle card; usage panel sums
include thinking tokens. Types are optional with `?? 0` defaults so a
brief web-before-backend rollout window cannot render NaN.
The original sessions table has expires_at TIMESTAMPTZ NOT NULL with no
default. Migration 000027 added the new columns but did not drop this one,
so CreateSession must still supply a value. Using AbsoluteExpiresAt.
Replace HS256 JWT access tokens with two opaque 32-byte random tokens
(access + refresh), both stored as SHA-256 hashes in sessions + Valkey.
Key changes:
- GenerateOpaqueToken() replaces JWT issuance; TokenService removed
- Sessions now carry access_token_hash, refresh_token_hash, family_id,
parent_session_id, access_expires_at, absolute_expires_at, last_used_at,
revoked_at — per migration 000027 (updated to add access_expires_at)
- Refresh rotation is atomic (UPDATE...RETURNING); reuse detection kills
the entire token family and returns auth.refresh_reuse_detected
- RequireAuth/OptionalAuth now take SessionLookup (Valkey→Postgres) instead
of *TokenService; sets session_id in context alongside user_id
- last_used_at is bumped on each request, throttled to writes >60s old
- AuthConfig{AccessTTL,RefreshIdleTTL,RefreshAbsoluteTTL} replaces JWT TTL env
vars (AUTH_ACCESS_TTL=30m, AUTH_REFRESH_IDLE_TTL=168h, AUTH_REFRESH_ABSOLUTE_TTL=720h)
- JWT_SECRET kept for AI-settings key derivation (drops from auth flow)
Forced logout on deploy (D3 behaviour); pre-launch so acceptable.
POST /admin/markets/:id/merge-plan now returns 202 + job_id immediately
and runs the Gemini advisor in a detached goroutine. Frontend polls
GET .../merge-plan/:job_id until done, with backoff up to 3 minutes.
Adds in-memory job registry (keyed map + RWMutex, 5-min TTL sweep) and
handler tests covering the full pending→done and error paths.
Merge-plan and research-plan both call Gemini which can take >60s.
The default gateway timeout was killing connections with 504.
- Web HTTPRoute: add /admin/ rule with 120s request+backendRequest timeout
- Backend HTTPRoute: add /api/v1/admin/markets/ rule with 120s timeout
- MergePlan handler: add 110s context deadline for graceful degradation
before the gateway cuts the upstream connection
Gemini returned field_merges as an array without structure constraint,
causing json.Unmarshal to fail with "cannot unmarshal array into Go struct
field of type map[string]mergeFieldDecision".
- Pass merge_advisor_schema.json via JSONSchema instead of bare JSONMode
- Add parseFieldMerges() that accepts both object and array LLM formats
- Validate target_id is one of the two input market IDs after parsing
- Fix schemaFromMap: minimum/maximum are supported by genai.Schema v1.54
H1: Drop empty string from enricher_schema.json category enum —
Gemini rejects enum[7]: cannot be empty (Error 400). Remove category
from required so the model can omit it when no category fits.
H2: Research-plan/apply client reads response as text before
JSON.parse; empty or HTML error bodies now surface the actual HTTP
status instead of crashing with "unexpected end of data".
I: Dedup UI for approved markets:
- DuplicatesPanel: LLM verdict pills (same/not-same, confidence),
llm_reason, per-candidate Merge-planen button
- MergeProposalPanel: summary, confidence, flags, per-field
decisions with editable source radio (a/b/combined), current
value context, confirm() before destructive apply
- Two SvelteKit proxy routes: merge-plan/ and merge-into/[targetId]/
- [id]/+page.svelte: wired with full state; navigates to survivor
after successful merge
- [id]/+page.server.ts: load duplicates for all non-merged editions
(was gated to status=rumored only)
- types.ts: DuplicateMarket gains llm_same/llm_confidence/llm_reason;
add MarketMergeProposal + MergeFieldDecision; add merged to
EditionStatus
MergeAdvisor calls Gemini with a German system prompt to propose how to merge
two duplicate market editions. It guards against confident non-duplicates via
ErrNotDuplicate (same=false AND confidence>0.5).
POST /:id/merge-plan generates a MarketMergeProposal (read-only).
POST /:id/merge-into/:target_id applies the merge: updates target fields,
marks source as status=merged with merged_into_id set, reparents discovered_markets,
and writes a market_merge_log audit row — all in one transaction.
AdminHandler gains advisor and updated constructor. VersionMergeAdvisor added
to pkg/ai versions.
Migration 000026 adds merged_into_id + merged_at to market_editions and
extends the status CHECK constraint to include 'merged'. FindSimilar now
excludes merged editions from candidates.
AdminHandler gains a SimilarityClassifier field; FindDuplicates enriches
the top 5 pg_trgm candidates with LLM same/confidence/reason verdicts.
simClassifier from routes.go is passed through to avoid a second instance.
schemaFromMap now logs a warning when keys genai.Schema ignores
(pattern, minLength, $ref, etc.) are present, keeping the workaround
visible. LLMEnricher skips Google Search grounding when total scraped
chars >= 1500, conserving free-tier quota on content-rich pages.
promptHashShort(system+"\x00"+user)[:12] computed on ErrSchemaViolation
and attached to ProviderError.PromptHash. research.go schema-violation
log now includes prompt_hash for cross-referencing ai_usage rows.
Migration 000024 adds prompt_version column + partial index.
PromptVersion plumbed through ChatRequest -> UsageEvent ->
buildUsageEvent -> settings INSERT/SELECT. Version constants
defined in ai/versions.go and wired at all three call sites.
- Add confidence scale (0.95-1.00 / 0.70-0.90 / 0.50-0.70 / 0.00-0.50)
with four annotated few-shot examples to the similarity system prompt
- Add two Ronneburg real-world pairs to similarity.json: descriptive-prefix
swap and low-trigram-overlap rename, both expected same=true
- Replace 3-example inline comment with 7-label taxonomy block so the
model knows all valid categories instead of guessing from partial hints
- Tighten description constraint to 60-220 chars with explicit word bans
- Mark opening_hours as a rough guide, not authoritative for booking
Adds ListEnrichedNeedingLLM to the Repository interface and RunLLMEnrichBacklog
to Service, then wires RunLLMEnrichBacklog into the post-crawl goroutine so
LLM enrichment runs automatically after every crawl without manual triggers.
Replaces JSONMode:true with an embedded enricher_schema.json so Gemini
returns structured output against a typed schema, preventing empty {} responses.
Adds an all-empty warning when the LLM returns a valid but blank payload.
Deterministic output is preferable for extraction and classification
tasks. Temperature=0.1 also enables the if-gate in gemini.go that
forwards the value to the Gemini API config.
Also add llm_enricher.go (renamed from mistral.go) with the temperature
field applied.
The researcher prompt incorrectly told the model to open/fetch URLs; it
only ever sees pre-fetched quellen[].text. Replace all "oeffnen" and
"aufgerufen" references with instructions to work from quellen[].
Add ZielJahr to research.Input and the JSON user-prompt payload so the
model has an explicit target year separate from recherche_datum (wall-clock
time of the request). Use ziel_jahr in the prompt instead of deriving the
year from recherche_datum. Fix the search query in the orchestrator to use
ZielJahr rather than RechercheDatum.Year().
Rename mistral.go → llm_enricher.go and mistral_test.go →
llm_enricher_test.go; update all test function names and stale model
strings (mistral-large-latest → gemini-2.5-flash-lite); drop Ollama
block from .env; mark superseded planning specs; update provider
references in planning docs and CLAUDE.md to Google Gemini.
- Add logo_url as a distinct DB column (migration 000023) and expose it
through model, DTOs, repository, service, and all frontend types
- Update KI-Recherche prompt and both JSON schemas: logo_url field rule,
clarified bild_url rule, hinweis now mandatory non-null (maxLength 200)
- imageURLReachable now also verifies Content-Type: image/* for both
bild_url and logo_url before surfacing suggestions
- MarketCard: image-first with cover style, logo fallback with contain
style, city-initial placeholder as last resort
- /markt/[slug]: hero section follows same image→logo→nothing precedence;
OG/JSON-LD updated accordingly
- Map view on search page: pagination hidden, map height increased to 600px
- Fix einstellungen Svelte warning: wrap showKeyInput init in untrack()
- Add missing= query param (description/image/website/location) to
AdminSearchParams; both AdminSearch and AdminSearchGrouped apply the
SQL condition
- Add has_description/has_image/has_website/has_location booleans to
AdminMarketSummary, populated in ToAdminSummary from existing Market fields
- Dropdown filter in the admin market list routes to the missing param
- Coloured dot indicators per row (amber=image, orange=desc, red=website,
purple=location) with title tooltips
SDK's modelFromMldev maps _self to tunedModelInfo for every model,
making the nil check always true and silently dropping all results.
Name-based filtering is the correct gate; tuned models are excluded
by the gemini- prefix requirement.
- Prompt now requires year verification before extracting any field
- Opening times and prices from prior years must be nulled with a hint
- imageURLReachable does a HEAD request (5s timeout) and strips the
image_url from research results when the resource returns 4xx/5xx
Description wasn't being applied because querySelector-then-assign runs before
Svelte's reactive flush of researchResult=null, which resets the textarea to
its initial market.description value. Fix: reactive state + exported setter
(same pattern as setHours/setAdmission).
Also add markt_name to felder in both schemas and the prompt so the LLM can
suggest a name correction. Name suggestions are gated to extraktion=direkt
(high confidence only) and guarded on the frontend with setName().
The beschreibung field was schema-required but absent from ## Felder,
causing the LLM to always return null. Add explicit extraction instruction.
Also reword the opening line which said "Keine Beschreibungstexte" —
contradicting the field we actually want.
On apply, append "KI-Recherche: DD.MM.YYYY HH:MM" to admin_notes so
there's a permanent audit trail of when research was run.
Gemini rejects requests that set both GoogleSearchRetrieval and
response_schema. The orchestrator already provides web content via
SearxNG + scraping, so grounding is unnecessary here.
factory.go: treat DB errors from GetGeminiAPIKey as "no key" and fall
back to the GEMINI_API_KEY env var instead of propagating the error
(which caused a panic/crash when migrations haven't been run yet).
gemini.go: ListModelNames returns a ProviderError when the client is
nil so that connected=false is reported correctly in GetAI instead of
the previous nil,nil→connected=true false positive.
+page.server.ts: catch fetch errors so a backend outage doesn't 500 the
whole page. +page.svelte: guard all data.ai access with {#if data.ai}
so the page renders an error banner instead of crashing on null access.
When a freshly-inserted discovered_market has a matched series, konfidenz
"hoch" (≥2 sources), and both start/end dates present, Accept() is called
inline with a nil reviewer (mapped to NULL reviewed_by) so the row goes
straight to accepted without manual review.
CrawlSummary gains auto_accepted counter; slog summary logs it.
MarkAccepted / Service.Accept now take *uuid.UUID for reviewer so nil
cleanly maps to NULL in the DB column (already nullable).
Replace the Mistral + Ollama AI stack with a single Google Gemini provider
backed by google.golang.org/genai. API key moves from env/Helm to the DB
(AES-256-GCM, key derived from JWT_SECRET via HKDF) so it can be rotated
via the admin UI without a pod restart.
New:
- pkg/crypto/secretbox — AES-256-GCM encrypt/decrypt for secrets at rest
- pkg/ai/gemini — GeminiProvider with grounding, structured output, usage
recording, and hot-reload (Reinitialize swaps client under mutex)
- pkg/ai/usage — UsageRecorder interface + UsageEvent struct
- domain/settings/store — DB-backed settings (model, grounding toggle, key)
- domain/settings/usage — UsageRepo implementing UsageRecorder; ai_usage table
- migrations 000021 (system_settings) + 000022 (ai_usage)
- settings API: GET /ai, POST /ai/key, POST /ai/model, POST /ai/grounding,
GET /ai/usage
- admin UI: 4-card settings page — provider status, model selector, grounding
toggle with quota, usage rollups + recent-calls table
Removed:
- pkg/ai/ollama, mistral_provider, ratelimiter (+ tests)
- Helm AI_API_KEY, AI_PROVIDER, AI_MODEL_COMPLEX, AI_AGENT_DISCOVERY,
AI_RATE_LIMIT_RPS env vars
Call sites set Grounded+CallType: research (true/"research"), enrich Pass B
(true/"enrich_b"), similarity (false/"similarity"). Integration test updated
to use a stub ai.Provider instead of a fake Ollama HTTP server.
Run CrawlEnrich + Nominatim geocoding in the background immediately
after a crawl discovers new rows. Manual triggers via the
/enrichment/crawl-all endpoint remain for backfills but are no longer
needed for fresh crawls.
