Student Onboarding & Registration — System Architecture

Audience: tech team (engineers building on top of this). Scope: the two ways a student gets into TAP LMS, plus the journey-tracking / flow-trigger layer that was built and then scrapped. Status: live (scratch + backend), with one deprecated layer documented in §4. Last updated: 2026-06-26 Companion doc: Student Onboarding — Operations Runbook (how to run this day-to-day).

0. TL;DR for a new engineer

What "registered" actually means. Creating a Student row is not the end of registration. A student is only truly registered once they start interacting with the TAP Buddy WhatsApp bot. Both paths below produce a Student record and put the contact into WhatsApp; the registration is "complete" only when the student responds to TAP Buddy. That first interaction — not the DB insert — is the real churn boundary (§6).

There are two onboarding paths and they barely share code:

Scratch (self-registration). The student sends a batch keyword over WhatsApp (the registration link pre-fills it), which starts the Glific flow. The flow collects their details and calls one synchronous API — tap_lms.api.create_student — which creates the Student + Enrollment inline. Because the student is conversing with the bot throughout, the data is complete and the student is already interacting. One student at a time, real-time.
Backend onboarding (bulk). Ops uploads a roster (a Backend Student Onboarding set), then runs a Desk page that turns those rows into Student + Enrollment records and Glific contacts in a two-phase background job (DB first, Glific second). The roster usually has gaps — fields the scratch conversation would have collected (e.g. gender, language) are sometimes missing — so those are confirmed/filled when the student later talks to TAP Buddy. After processing, every student in the set is in a Glific collection (group), and the onboarding flow is triggered in Glific against that collection/group id to start the TAP Buddy conversation. Deeply documented in docs/backend-onboarding-process-flow.md; this doc summarises and cross-references it.

A third layer — in-LMS per-stage journey tracking + an Onboarding Flow Trigger page (status-filtered triggering and progress tracking, driven from TAP LMS) — was built on top of both paths and scrapped (§4). Note the distinction: triggering the flow still happens for backend (directly in Glific, by collection/group id); what was abandoned is the TAP-LMS-side stage machinery and the tracking/nudge/escalation automation around it. It failed for the same reason the backend path needed a throughput rewrite: Glific HTTP calls are slow and fail, and the original code had no timeout, no retry, and no dead-letter (§5).

1. Domain model (the doctypes)

DocType	Role	Notes
Student	The canonical student record.	`Student.name` (e.g. `ST00051383`) is the student ID (L-031). Display name is `Student.name1`. Holds `phone`, `gender`, `school_id`, `grade`, `language`, `glific_id`, and an `enrollment` child table.
Enrollment (child of Student)	One term enrollment.	`batch`, `course` (a Course Level), `grade`, `school`, `date_joining`. `hash`-named since CR-2026-06-19 (removed `ER`-counter lock contention).
Batch	The cohort.	`active` flag and `regist_end_date` gate scratch registration.
Batch onboarding	Maps a `batch_skeyword` → `school`, `batch`, `kit_less`.	The lookup table that makes a WhatsApp keyword resolve to a school + batch. Keyword auto-generated on insert (`batch_onboarding_utils.generate_unique_batch_keyword`).
Backend Student Onboarding ("the set")	A bulk upload batch.	`status` (Draft→Processing→Processed/Failed), `student_count`, `processed_student_count`.
Backend Students (child of the set)	One staged row per student to onboard.	`processing_status` (Pending→Success/Failed), `glific_sync_status` (pending→synced/failed), `student_id` (→ created Student).
Course Verticals / Course Level / Stage Grades / Grade Course Level Mapping	Course-level resolution inputs.	Both paths resolve grade + vertical (+ student type + kit-less) → a Course Level. See §3.
OnboardingStage / LearningStage / StudentStageProgress	The journey-tracking layer.	Deprecated for onboarding (§4).

Student, Enrollment, and the course-level reference tables are shared by both paths. Batch onboarding is scratch-specific; Backend Student Onboarding / Backend Students are backend-specific.

2. Path A — Scratch (self-registration)

Entry point: POST /api/method/tap_lms.api.create_student — app/tap_lms/api.py:347, @frappe.whitelist(allow_guest=True).

Flow

Student clicks WhatsApp link (carries batch_skeyword)
        │
        ▼
Glific registration flow  ── collects: name, phone, gender, grade, language, vertical
        │                     (glific_id already exists — the contact is in Glific)
        ▼
POST create_student(api_key, student_name, phone, gender, grade,
                    language, batch_skeyword, vertical, glific_id)
        │
        ▼  (synchronous, inline — no background job)
   1. authenticate_api_key(api_key)                      → 202 "Invalid API key"
   2. all 8 fields present?                              → 202 "All fields are required"
   3. Batch onboarding lookup by batch_skeyword          → 202 "Invalid batch_skeyword"
   4. Batch.active? regist_end_date not passed?          → 202 "not active" / "registration ended"
   5. Course Verticals lookup by label (name2)           → 202 "Invalid vertical label"
   6. Student exists for this glific_id?
        ├─ yes & name+phone match → update grade/language/school
        └─ else                   → create_new_student(...)
   7. get_course_level_with_mapping(vertical, grade,     → 202 "Course selection failed: …"
        phone, name, kitless)
   8. append Enrollment {batch, course, grade, school}; save()
        │
        ▼
{ "status": "success", "crm_student_id": "ST…", "assigned_course_level": "…" }

Properties that matter

Synchronous and inline. No queue, no retry. The whole thing runs inside the Glific webhook request. The student is waiting on WhatsApp.
It does not create a Glific contact. The contact already exists (Glific is the caller); create_student only stores the supplied glific_id on the Student. Contrast with the backend path, which does create/optin Glific contacts.
Errors return HTTP 202 with {"status":"error","message":...}. Glific is expected to show a retry prompt. (202 is used as a soft-error signal, not 4xx/5xx — keep this in mind if you ever put a real status-code-based monitor in front of it.)
Idempotency is by glific_id. A re-call with the same glific_id + matching name/phone updates rather than duplicates; a mismatch creates a new Student. There is no enrollment-idempotency guard here (unlike the backend path's L-073) — a repeated successful call can append a second Enrollment for the same batch.
Logging is effectively off. Most frappe.log_error calls in this function were replaced by print or removed (see the # REMOVED: Problematic logging comments). This is the single biggest obstacle to a churn analysis: scratch failures are not durably recorded.

Supporting scratch endpoints (all `allow_guest=True`, in `api.py`)

Endpoint	Purpose
`verify_batch_keyword()`	Validate a keyword before registration; returns `tapschool:<skeyword>` form.
`list_batch_keyword(api_key)`	List a school's available batch keywords (for Glific prompts).
`get_batch_keywords_by_phone(api_key, phone_number)`	Look up the batches a phone is already enrolled in.
`glific_get_courses()` / `glific_list_grades(vertical)`	Populate vertical/grade pickers in the Glific flow.
`get_course_level_api(...)`	Course-level resolution exposed to Glific.

3. Course-level resolution (shared sub-flow)

Both paths must turn (vertical, grade, student_type, kit_less) into a Course Level before the enrollment can be written. "Without course level the system breaks" (CR-004), so the chain must stay intact; optimise with caching/indexes only.

determine_student_type(...)        New vs Old — has this phone+name enrolled in this vertical before?
        │
        ▼
get_course_level_with_mapping(...)  1. Grade Course Level Mapping for current academic_year
                                    2. → mapping with academic_year unset (flexible)
                                    3. → get_course_level_original(...)  (Stage Grades fallback,
                                         with kit-less → non-kit-less fallback)

Scratch uses get_course_level_with_mapping (api.py).
Backend uses get_course_level_with_validation_backend → get_course_level_with_mapping_backend → determine_student_type_backend (backend_onboarding_process.py). Same shape, backend-suffixed.
Academic year runs April→March; "2025-26" form.
Index idx_gclm_lookup on Grade Course Level Mapping (academic_year, course_vertical, grade, student_type, is_active) collapses the lookup to one probe (CR-004).

4. Path C — Journey tracking + Onboarding Flow Trigger (BUILT, THEN SCRAPPED)

This is the layer the project set out to build last year to measure registration churn and automate nudges/escalation, and the one that was abandoned. Documenting it so the next attempt starts from the real reasons it failed.

4.1 What it was supposed to do

Two designs were written (see the Notion docs Scratch Student Registration API and Backend Onboarding and Student Journey Tracking):

Journey tracking. Wrap registration in instrumentation. Glific flows fire webhooks at flow_started, flow_step_completed, flow_completed, assessment_submitted → tap_lms.journey.api.track_interaction. The system records StudentStageProgress (status: not_started / assigned / in_progress / completed / incomplete / skipped) against OnboardingStage and LearningStage rows, plus LearningState / EngagementState. A daily sweep catches missing records. A monitoring read API, get_students_by_glific_group(group_id, stage_id, stage_type, status, …), was specified to segment students for follow-up — note this endpoint was never actually implemented in code (it appears only in the Notion design), which is itself a sign of how far the tracking layer got before being abandoned.
Onboarding Flow Trigger. A Desk page (onboarding_flow_trigger) that takes an onboarding set + an OnboardingStage + a target student status and fires the stage's Glific flow — either a group flow (startGroupFlow) or individual flows (per-student start_contact_flow) — then updates StudentStageProgress to assigned. The intended use was to nudge "not_started" students and escalate "incomplete" ones.

What actually replaced it operationally. Backend onboarding still triggers the TAP Buddy flow after processing — but from Glific, against the set's collection/group id, not through this TAP-LMS stage page. Each set creates a GlificContactGroup (group_id = the Glific collection id, linked via backend_onboarding_set); Phase 2 adds every contact to it; an operator (or Glific) then starts the group flow on that group_id. So triggering survived; the in-LMS stage tracking, status filtering, and nudge/escalation automation did not.

4.2 What exists in code today

Component	File	State
`track_interaction`, `update_student_stage`	`tap_lms/journey/api.py`	Deprecated 2026-05-28 — return a structured deprecation envelope (`journey/_deprecation.py`), no side effects.
`get_next_content`, `complete_content`, `start_quiz`, `submit_answer`, `get_student_progress_overview`, …	`tap_lms/journey/student_progression.py`, `student_api.py`, `student_preferences_api.py`	Deprecated 2026-05-28 — superseded by `tap_lms/summer_program/*`.
`trigger_onboarding_flow`, `trigger_group_flow`, `trigger_individual_flows`, `get_onboarding_progress_report`	`tap_lms/tap_lms/page/onboarding_flow_trigger/onboarding_flow_trigger.py`	Still present, hardened (CR-025), but not part of the live onboarding flow. It can fire a Glific flow and write `StudentStageProgress` locally, but the webhook surface that would record student responses (`track_interaction`) is deprecated — so progress can't round-trip. As of 2026-06-26 this page is not in use; whether to revive in-LMS triggering is parked pending a stakeholder workflow decision — treat it as a deferred decision, not a closed deprecation (the tracking webhooks above are closed; this page is not).
`update_incomplete_stages`	same file (`:864`)	⚠️ *Not fully dormant — still registered as an active daily* scheduled job (`hooks.py` → `scheduler_events["daily"]`, `:69`). In practice it is a no-op: it only flips `OnboardingStage` `StudentStageProgress` rows from `assigned`→`incomplete` after 3 days, and nothing in the live path creates `assigned` `OnboardingStage` rows** — backend Phase 1 writes them `not_started`, the live SP path writes `LearningUnit` (not `OnboardingStage`) rows, and the only `assigned`+`OnboardingStage` writer is the dormant trigger page itself (plus the deprecated, now-unreachable `journey/api.py`). So the job runs daily and matches zero rows unless an operator manually fires the trigger page.
`OnboardingStage`, `LearningStage`, `StudentStageProgress`, `StudentOnboardingProgress` doctypes	`tap_lms/tap_lms/doctype/…`	Schema present; used only by the dormant trigger/report code.

So: the trigger machinery is half-alive, the tracking webhooks are dead. Net effect for the product: there is no working closed loop for automated nudges/escalation on onboarding. (The tracking doctypes are not dead, though: backend Phase 1 still creates a StudentStageProgress (not_started) + LearningState + EngagementState per student, and the daily update_incomplete_stages job still runs against OnboardingStage rows — the records get written every run, they just never advance past not_started because no live writer ever sets them to assigned.)

4.3 Why it was scrapped (the real reasons)

Glific API timeouts and failures. The original Glific calls used a bare requests.post with no timeout, no shared session, no retry, and no dead-letter. A single slow/hung Glific endpoint blocks the worker. This is not theoretical — CR-004 records a live 2026-05-31 incident where a no-timeout Glific requests.post wedged a background worker and required manual operator recovery. The same fragility sat under trigger_group_flow / trigger_individual_flows.
Per-student fan-out at WhatsApp speed. Individual flow triggers loop students in batches of 10 with time.sleep(2) between batches to dodge rate limits — i.e. the design assumed it would be slow and still risked 429s. Group flows reduce calls but then "TAP LMS triggers the flow for ALL students in the group" with no status filtering on the Glific side, which defeats targeted nudging.
Couldn't reliably automate nudges & escalation. Because responses didn't round-trip dependably (webhook failures, no retry, lossy logging), StudentStageProgress drifted from reality, so "who is stuck / who to nudge" couldn't be trusted — which was the whole point.
Superseded. The live cohort moved to the summer_program state machine (T0–T25, per-PE dispatcher, retry/DLQ, fast-path APIs), which is where escalation/nudge logic was rebuilt properly. The journey surface was formally deprecated 2026-05-28 rather than maintained in parallel.

4.4 What a rebuild must handle (for whoever revives nudges/escalation)

Every Glific call through a shared requests.Session with an explicit timeout, wrapped in retry + dead-letter (the CR-004 / CR-025 pattern). No bare requests.post.
Treat WhatsApp delivery as eventually-consistent: a state machine with retryable transitions, not a synchronous "trigger then assume assigned."
Durable, structured logging on the registration path itself (scratch create_student currently logs almost nothing — fix this first or churn stays unmeasurable).
Reuse summer_program's escalation/dispatcher rather than reanimating journey/*.

5. Backend processing & the throughput work (summary)

Full detail: docs/backend-onboarding-process-flow.md and CR-004. Summary here so this doc stands alone.

5.1 Two-phase pipeline

Trigger: process_batch(batch_id) (backend_onboarding_process.py) — TAP Admin only, concurrency-guarded (_onboarding_job_is_active), sets the set to Processing, enqueues process_batch_job on the long queue (2h timeout).
Phase 1 — DB only (fast). Per student, inside a Postgres savepoint with serialization retry: resolve course level, process_student_record (create/update Student + append Enrollment + init states), mark glific_sync_status="pending", processing_status="Success". Zero Glific calls (AC-1).
Phase 2 — Glific sync (slow, retryable). One sync_student_to_glific job per pending/failed row on long: process_glific_contact (lookup/create/optin/group-add), write glific_id back, set synced. Transient errors retry ≤3 then dead-letter to Error Log with glific_sync_status="failed" — the Student already exists, only the Glific link is pending (L-072).

A set is done only when both phases finish — Phase 2 is the bottleneck. Phase 2 also puts every contact into the set's Glific collection/group (create_or_get_glific_group_for_batch → GlificContactGroup.group_id). After processing, the TAP Buddy onboarding flow is triggered in Glific against that group_id — that flow is what starts the student interaction (and lets the bot collect any roster fields that were missing). Until the student responds to it, registration is not "complete" (§0, §6).

5.2 What made it faster (CR-004, approved 2026-05-31)

The original single-pass loop did ~5 sequential Glific round-trips per new student inline, on a single worker, with an unindexed tabStudent.phone scan that degraded ~O(n²) over a 75,000-student run. The fixes, in order of impact:

Lever	Change
Phase split	Move all Glific I/O out of the DB loop into independently-retryable Phase-2 jobs. DB side now runs at thousands/min.
Session + timeout + retry/DLQ	Shared `requests.Session`, explicit 10 s timeout, retry≤3, dead-letter. Priority-zero fix after the worker-wedge incident. Also fixed the unbound-`glific_contact` `NameError` that mismarked transient failures as `Failed`.
Indexes	`idx_student_phone` (dedup) and `idx_gclm_lookup` (course-level mapping). `VACUUM ANALYZE` after.
Per-job caching	Memoise School / TAP Language / Course Vertical / Course Level reference lookups per run.
Worker parallelism	`background_workers` 1 → 4 so sets run concurrently. Bounded 4–6 to avoid Glific 429s (retry/DLQ absorbs the rest).
Hash-named Enrollment	CR-2026-06-19 — drop the shared `ER` counter lock so parallel workers don't serialize on it.

Follow-up (CR-DRAFT-backend-onboarding-phase2-throughput.md): drop the redundant phone lookup (B1) and batch group-adds 500:1 (B2) to roughly halve Phase-2 wall-clock.

5.3 Resilience invariants (don't regress)

AC-1 (no Glific in Phase 1), L-069 (gender fill-only), L-073 (enrollment idempotency on student_id+batch), L-029 (archetype/experiment_arm upstream-supplied), L-072 (done = both phases), L-039 (db.set_value on Student.glific_id is safe), H1 (per-student savepoint, never blanket rollback).

6. Where churn lives (so the next analysis can target it)

Completion bar: registration is complete only when the student interacts with TAP Buddy. A created Student who never replies is churned, not registered. Every row below ends at that bar.

Path	Funnel to the completion bar	Where a student is "lost"	Currently measurable?
Scratch	sends keyword → flow starts → data collected → `create_student` → keeps interacting with TAP Buddy	Abandons the Glific flow before `create_student`; or `create_student` returns a 202 error (invalid keyword, batch closed, course-level miss, missing field) and the student doesn't retry. (Less of a "first interaction" gap here — the student is already conversing — more an in-flow abandonment gap.)	Poorly. `create_student` logging was stripped; the in-flow funnel lives only in Glific analytics. The deprecated journey layer was meant to capture exactly this and never shipped.
Backend	roster uploaded → Phase 1 (Student created) → Phase 2 (Glific contact + added to collection) → flow triggered on `group_id` → student responds to TAP Buddy	(a) Phase-2 sync dead-letters (`glific_sync_status="failed"`) → never reachable on WhatsApp; (b) reachable, flow triggered, but the student never responds to TAP Buddy — the dominant churn bucket, and the one bulk onboarding is most exposed to since the student had no prior touch; (c) responds but the bot can't complete because roster data was missing and the conversation stalls.	Partially. `processing_status` + `glific_sync_status` cover up to the trigger. "Never responded to TAP Buddy" is not tracked in TAP LMS since journey tracking was scrapped — it's only visible in Glific flow analytics.

Implication for a quantitative churn study: - Up to the trigger boundary, backend is recoverable from glific_sync_status + Error Logs. - The decisive metric — did the student interact with TAP Buddy — is not in TAP LMS today. It lives in Glific flow analytics, or requires re-instrumenting an inbound webhook (the well-built version of what track_interaction was meant to be). - Backend data gaps (missing gender/language/etc. that scratch collects conversationally) are a second-order churn driver: if the bot has to ask for too much, or routing depends on a field that's blank, the conversation stalls. Worth auditing which roster fields are most often missing.

Closing this measurement gap (a reliable "first TAP Buddy interaction" event) is the prerequisite before churn can be quantified end-to-end.

7. Endpoint reference (onboarding domain)

Endpoint	Path	State
Scratch register	`tap_lms.api.create_student`	Live
Verify keyword	`tap_lms.api.verify_batch_keyword`	Live
List keywords	`tap_lms.api.list_batch_keyword`	Live
Keywords by phone	`tap_lms.api.get_batch_keywords_by_phone`	Live
Backend trigger	`…backend_onboarding_process.process_batch`	Live (TAP Admin)
Backend job status	`…backend_onboarding_process.get_job_status`	Live
Flow trigger (in-LMS)	`…onboarding_flow_trigger.trigger_onboarding_flow`	Unused — decision pending. Live triggering is done in Glific against the set's collection `group_id`, not here; reviving in-LMS triggering is parked on a stakeholder workflow decision (see §4.2).
Onboarding progress report	`…onboarding_flow_trigger.get_onboarding_progress_report`	Dormant
Journey tracking	`tap_lms.journey.api.track_interaction`, `update_student_stage` & all `journey.*`	Deprecated 2026-05-28 (return a deprecation envelope; originals renamed `_DEPRECATED_*_original`)
Journey group monitor	`get_students_by_glific_group`	Never implemented — specified in the Notion design only; no code exists.

8. Pointers

Backend deep-dive: docs/backend-onboarding-process-flow.md
Throughput rationale + AC: docs/change-requests/CR-004-backend-onboarding-throughput.md, CR-DRAFT-backend-onboarding-phase2-throughput.md
Glific hardening: CR-025-glific-sync-401-hardening.md
Replacement journey system: docs/architecture.md (summer_program §) and tap_lms/summer_program/*
Source: api.py (scratch), tap_lms/page/backend_onboarding_process/ (backend), tap_lms/page/onboarding_flow_trigger/ + tap_lms/journey/ (scrapped layer)