Leads End-to-End Testing

The leads platform is four Volkanos modules working as one pipeline: leads, siteintel, communicator and notifications, listed with their packages in the module catalog, each documented in its own repo. One reference scenario proves they fit together: a company imported from a CSV row gets a site audit, an AI draft, a human review, a sandboxed send, a reply, a stage change and an escalation email, with nobody touching the database. If this scenario is green, the funnel works. The funnel needs the Entirius AI toolbox, which is not public: without credentials from the Entirius team the AI steps (intel analysis, drafts) fail and the scenario stays red.

Everything runs in entirius-zeno against the Emporium test package. Mail never leaves the stack: every message lands in a GreenMail sandbox.

Three Modes

The same scenario, the same seed, three ways to drive it.

Mode	Who acts	How	When
A · BDD	nobody: the admin API v2	make bdd TAGS=@funnel	every backend change, CI
B · e2e automated	Playwright in the admin CMS, phone emulation then desktop	make e2e-funnel	CMS changes, nightly
C · AI-tester acceptance (optional)	an AI session with the ux-tester role clicks through the CMS like a human tester and writes a report with screenshots	make e2e-accept	CMS changes, releases

Mode A asserts the contract. Modes B and C prove a person can walk the same path in the CMS without dead ends. Mode B builds its own state through the API, so it runs twice on one seed (iPhone 14, then desktop); the page objects, the one-shot tags and the determinism notes are in the Emporium test package, docs/e2e-leads-funnel.md.

Prerequisites

make dev             # stack with repos/ mounted: service, worker, beat, GreenMail
make toolbox-check   # AI toolbox answers and exposes only `fake` models
make migrate
make seed            # ends with SEED OK; purges GreenMail, checks the four modules are registered

The AI toolbox runs outside zeno. Start it before make seed and keep it up. make toolbox-check fails on any real model, so a green check means every draft comes from a fixture, not from a paid call.

Mode C is optional: it needs Claude Code and the dev-runner profiles of the zeno repo (make runner-init), the maintainers’ agent tooling, which is not required to contribute. Modes A and B need neither.

Modes B and C add the CMS and, for C, the AI-tester profile:

make mail                   # GreenMail, if the stack was not started with it
make cms-dev                # admin CMS on :8180 from repos/pwa/entirius-pwa-cms
make bdd TAGS=@funnel       # mode A first, on the fresh seed
make e2e-funnel             # mode B
make runner-init            # once: registers the ux-tester role
make e2e-accept             # mode C; report and screenshots in .runner/accept/<ts>/

make e2e-accept exits non-zero when report.md is missing or lists anything under ## Blockers.

The Scenario

The funnel company is example-shop-1.test with two contacts, [email protected] and [email protected] (PL, legitimate interest), addresses no other feature uses. The channel is default-europe and the channel clock is set to Monday 10:00 of a holiday-free reference week. Every step starts by remembering the GreenMail count of the funnel’s own mail (to its contacts, or naming them or the company in the subject), so the mailbox checks are deltas within the step and hold inside the full make bdd run.

Step	What happens	Assertion	Visible in A	Visible in B/C
0	Setup	munin lists leads, communicator, siteintel, notifications; PATCH channel/ to sandbox answers 200 and GET channel/ reads sandbox	api/munin/v2/, communicator/…/channel/	none
1	CSV import of package/leads--funnel.csv	batch done, created_count 1 (the second row matches the same company), one company in stage new, Activity import “import created”	leads/…/test/import-now/, imports/<id>/, companies/	Leads board, column new
2	Stage rule new requests an audit	rule run fired, Activity intel “audit requested”; audit partially_completed: lighthouse (PageSpeed Insights) completed with processed strategies, urlscan completed, heuristic failed	leads/…/rule-runs/, activities/, siteintel/…/audits/<id>/	Company › Intel
3	Intel analysis, recipient pick and draft	Activity intel “intel analysed”, 3 hooks and a platform, Activity draft “draft review_required”, exactly one Activity rule “recipient picked” with a contact_id, a draft about the company in the review queue	leads/…/companies/<id>/, activities/, communicator/…/review/	Inbox, 1 to review
4	Review	draft accepted: approved with scheduled_at	review/<id>/accept/	Inbox empty, scheduled time shown
5	test/send-due/	funnel mail count +1; one mail to anna with [SANDBOX] subject, multipart/alternative with a communicator- Message-ID, text and html parts with the legal footer	communicator/…/test/send-due/, GreenMail REST	Thread shows sent
6	Reply	reply fixture from anna injected with In-Reply-To of the sent mail; test/poll-now/ → thread replied, reply of kind reply matched by header	threads/<id>/, replies/	Reply under the sent mail
7	Leads reacts	stage replied, Activity reply “reply received”, two high notifications for the thread (communicator and leads); test/run-escalation/ → mail “Reply from Example Shop 1” and funnel mail count +2	companies/<id>/, notifications/, GreenMail REST	Leads board column replied
8	Negative	manual communicate/ for a do_not_contact company: 409, Activity blocked “blocked: do_not_contact”, no draft in the review queue, test/send-due/ adds no mail	companies/<id>/communicate/, review/	none

A company created by the import starts in new without a stage change, so mode A runs the stage rules for step 2 through test/evaluate/. The recipient pick only runs when a company has more than one eligible contact, which is why the funnel CSV carries two; the fake model picks the first candidate, so the mail goes to anna.

Three results look like gaps and are not:

• No notification for a new draft. The review queue is the signal; notifications start with the reply.
• heuristic fails. Lighthouse and urlscan read recordings, but the heuristic source fetches the site itself and example-shop-1.test has no synthetic site. The audit ends partially_completed by design and the analysis runs on the two sources that succeeded.
• Step 8 answers 409 with no draft. The outreach gate runs on every path, the manual one included, and refuses a do_not_contact company before a message exists.

Worker-side effects (analysis, drafts, stage reactions) are asynchronous; the BDD steps poll them. Beat does not drive mode A: the scenario calls the development-only test endpoints (test/send-due/, test/poll-now/, test/run-now/<audit_id>/, test/run-escalation/) instead of waiting five minutes. The beat container runs the real schedule for modes B and C.

One-Shot

@funnel consumes the import, the audit, the draft and the reply. A second run on the same database finds an existing company and no new draft, and fails. Re-seed before every run:

make seed && make bdd TAGS=@funnel

The same applies to @leads-oneshot and @communicator-oneshot scenarios in the full suite.

When It Fails

The failing scenario names the step (Funnel 3 …). Check in this order:

1. docker compose logs worker: rules, analysis and stage reactions run here. A task that never started means the worker does not consume its queue, or it runs stale code (Celery has no autoreload, restart it).
2. docker compose logs beat: only for modes B and C; it lists the schedule at start.
3. GreenMail REST at http://localhost:8380/api/user/sandbox/messages/INBOX: what actually arrived.
4. api/leads/v2/admin/default-europe/activities/?company=<id>: the company timeline says skipped: … or blocked: … with the reason when a rule refused to act.
5. docker compose logs service: API errors and toolbox calls.

What Is Not Asserted

The draft text comes from a fake model that answers by tag, so the scenario checks formal rules only: parts, headers, footer, stages, activities. Prompt quality is not judged here. The token tracker (two entries tagged leads.analysis and leads.pick_recipient) is not asserted in mode A because the toolbox runs outside zeno; the staging canary checks it with a real model.

Staging Canary

Zeno proves the pipeline with a fake model and a local mailbox. Before any channel goes live, the same scenario runs once more in mode C against staging:

• Volkanos on staging runs the released package versions with ENVIRONMENT=staging.
• The canary channel stays in sandbox mode with a real sandbox mailbox (IMAP, app password). A channel cannot enter sandbox without a mailbox, so configure the mailbox first.
• The staging toolbox channel points at a real catalogue model and does not expose fake. make toolbox-check does not apply here. It is red on any real model; check the catalogue by hand.
• The channel carries its clause texts, a suppression list, a send policy and one company on a domain the team owns.

make e2e-accept reads its URLs from the local zeno stack (.env, make urls); a canary run points the ux-tester at the staging CMS and API instead. Nothing is injected in the canary: a person answers the sent mail from the sandbox mailbox for step 6. It is accepted when the report has no blockers and a person has read the generated copy. Secrets stay in the staging settings, never in the report or its screenshots.

Next

Security