Product & Technology

SOP 0: Technology Stack Overview

Purpose

Understand the full technology stack of a Regenesis pod — hardware, software, and infrastructure — so that any team member can quickly orient themselves on how the system works end-to-end.

Frequency

Reference document — review quarterly or when stack changes.

Owner

Tav (Product Director), Sarid (CTO)

Maintainer

Wina

Time Needed

N/A (reference document)

The Regenesis Pod Technology Stack

1. Hardware Layer

\[table-embed:1:1 Component | 1:2 Details | 2:1 ESP32 Modules | 2:2 9 per pod — each controls a specific subsystem (lighting zones, audio zones, sensor inputs) | 3:1 Amplifier | 3:2 JAB5 — drives the pod's speaker array | 4:1 PC Control Unit | 4:2 Central brain — runs the control application, coordinates all ESP32 modules | 5:1 Sensors | 5:2 Biofeedback sensors (type TBD — confirm with Matt), environmental sensors (temp, humidity) | 6:1 Power | 6:2 UPS-backed power supply (see Manufacturing SOP 07 for UPS procedures) |\]

2\. Software Layer

\[table-embed:1:1 Component | 1:2 Details | 2:1 Embedded Firmware | 2:2 Runs on each ESP32 module — controls hardware I/O, publishes status via MQTT | 3:1 Control Application | 3:2 Runs on the pod PC — orchestrates sessions, receives sensor data, drives audio/lighting | 4:1 MQTT Messaging | 4:2 All inter-component communication uses MQTT (publish/subscribe model) | 5:1 Firmware Repository | 5:2 Stored in MinIO — versioned firmware binaries for OTA or manual flashing |\]

3\. Infrastructure Layer

\[table-embed:1:1 Component | 1:2 Details | 2:1 VPN | 2:2 Wireguard — secure remote access to each pod for maintenance and updates | 3:1 MQTT Broker | 3:2 VerneMQ — central message broker for all pod communications | 4:1 MinIO | 4:2 Object storage — hosts firmware binaries, configuration files | 5:1 Monitoring | 5:2 TBD — confirm with Sarid what monitoring is in place |\]

4\. Communication Flow

Sensors -> ESP32 Modules -> MQTT -> Control PC -> Audio/Lighting Output

^ |

Firmware (MinIO) Session Logic

^ |

VPN (Wireguard) User Interface

Cross-References

Manufacturing SOP 04 — ESP32 Flash & Firmware Loading (hardware flashing steps)

Manufacturing SOP 05 — Quality Control & Testing (pod QC including MQTT verification)

Manufacturing SOP 07 — UPS System (power management)

Product & Technology SOP 1 — Firmware Release Process

Product & Technology SOP 4 — MQTT System Health Check

TBD Items

\[table-embed:1:1 Item | 1:2 Who Needs to Confirm | 2:1 Biofeedback sensor types and specifications | 2:2 Matt | 3:1 Current monitoring/alerting stack | 3:2 Sarid | 4:1 Control application tech stack (language, framework) | 4:2 Sarid | 5:1 Network architecture diagram | 5:2 Sarid |\]

Done When

New team member can describe the full pod stack (hardware, software, infrastructure) after reading this document

All TBD items have been confirmed and filled in

Diagram matches actual deployed architecture

Can't find a component mentioned here: Check with Sarid or Tav — the stack may have changed since last review

Discrepancy between this doc and reality: Update this SOP immediately and notify Wina

New component added to the stack: Add it here, update cross-references, notify all SOP owners in this department

SOP 1: Firmware Release Process

Purpose

Safely release new firmware to production Regenesis pods, ensuring quality gates are passed, rollback is possible, and all stakeholders are informed.

Frequency

As needed — each firmware release cycle.

Owner

Sarid (CTO), Fahmy

Maintainer

Wina

Time Needed

~2-4 hours per release (testing + deployment + verification)

Steps

Phase 1: Pre-Release

Confirm the release scope — review the changelog with Sarid. What modules are affected? What bugs are fixed? What features are added?

Version numbering — apply the agreed version convention (TBD — confirm with Sarid: semantic versioning? date-based? module-specific?)

Run unit tests — all unit tests must pass with zero failures

Run integration tests — test firmware on a development pod (not production) to verify inter-module communication

QA sign-off — Sarid or designated QA person reviews test results and signs off in ClickUp task comments

Document known issues — list any known limitations or edge cases in the release notes

Phase 2: Upload to Repository

Build the release binary — compile firmware for target ESP32 modules

Upload to MinIO — place the binary in the firmware repository with correct version folder structure

Verify the upload — download the binary from MinIO and confirm checksum matches the build output

Tag the release in version control — create a git tag matching the firmware version

Phase 3: Deployment

Notify affected pod owners — inform them of scheduled update window (if remote/OTA)

Deploy to target pod(s) — follow one of:

OTA deployment (if available): trigger update via control application

On-site deployment: follow Product & Technology SOP 3 (Pod Software Update — On-Site)

Manual flash: follow Manufacturing SOP 04 (ESP32 Flash & Firmware Loading)

Verify deployment — confirm all 9 ESP32 modules are running the new firmware version

Run post-deployment health check — follow Product & Technology SOP 4 (MQTT System Health Check)

Phase 4: Post-Release

Monitor for 24 hours — watch for error reports, module disconnections, or unexpected behavior

Update firmware version tracker — record which pods are running which firmware version

Close the release task in ClickUp — attach release notes, test results, and deployment confirmation

Rollback Procedure

If issues are found after deployment:

Assess severity — is the pod non-functional (Critical) or degraded (High/Medium)?

For Critical: immediately roll back to the previous firmware version stored in MinIO

For High/Medium: document the issue, create a bug ticket (see SOP 2), schedule a fix

Roll back steps:

Download previous firmware version from MinIO

Flash affected modules (Manufacturing SOP 04)

Verify all modules responding (SOP 4 — MQTT Health Check)

Notify stakeholders of rollback

Cross-References

Manufacturing SOP 04 — ESP32 Flash & Firmware Loading (actual flashing steps)

Manufacturing SOP 05 — Quality Control & Testing (post-update QC)

Product & Technology SOP 0 — Technology Stack Overview

Product & Technology SOP 2 — Software Bug Reporting (if issues found)

Product & Technology SOP 3 — Pod Software Update On-Site

Product & Technology SOP 4 — MQTT System Health Check (post-deployment verification)

TBD Items

\[table-embed:1:1 Item | 1:2 Who Needs to Confirm | 2:1 Version numbering convention (semantic vs date-based vs other) | 2:2 Sarid | 3:1 OTA update capability — is this available or planned? | 3:2 Sarid | 4:1 MinIO folder structure for firmware versions | 4:2 Sarid | 5:1 Automated test suite — does one exist? What framework? | 5:2 Sarid |\]

Done When

All tests pass (unit + integration)

QA sign-off recorded in ClickUp

Firmware binary uploaded to MinIO with correct version

All target pods updated and verified (9/9 modules responding)

24-hour monitoring period passed with no Critical issues

Release notes and version tracker updated

Test failure during pre-release: Do NOT proceed. Fix the failing test or the code, then restart from step 3

Upload to MinIO fails: Check MinIO connectivity and credentials. Contact Sarid if access issues

Module not responding after update: Check MQTT health (SOP 4). If unresponsive, roll back that module to previous firmware

Multiple modules failing post-update: Initiate full rollback immediately. Do not wait for monitoring period

Version mismatch detected: Verify which version each module is actually running. Re-flash mismatched modules

SOP 2: Software Bug Reporting

Purpose

Report software bugs in a way that helps developers fix them quickly, with consistent severity classification and clear response time expectations.

Frequency

As needed — whenever a bug is discovered.

Owner

Sarid (CTO)

Maintainer

Wina

Time Needed

10-15 minutes to file a complete bug report.

Steps

1. Identify and Classify the Bug

Before reporting, confirm it is actually a bug (not a feature request, not user error, not a known issue).

Priority Levels:

\[table-embed:1:1 Priority | 1:2 Definition | 1:3 Examples | 1:4 Response SLA | 2:1 Critical | 2:2 Pod is non-functional. Customer cannot use the pod. | 2:3 Pod won't start, all audio dead, safety system failure | 2:4 4 hours | 3:1 High | 3:2 A major feature is broken but pod is partially usable. | 3:3 One lighting zone dead, biofeedback not recording, session crashes mid-way | 3:4 24 hours | 4:1 Medium | 4:2 Feature works but is degraded or inconsistent. | 4:3 Delayed sensor readings, occasional MQTT timeouts, UI glitch | 4:4 1 week | 5:1 Low | 5:2 Cosmetic or minor issue. Pod fully functional. | 5:3 Incorrect label in UI, log formatting issue, minor visual bug | 5:4 Next sprint |\]

2\. Gather Required Information

Collect ALL of the following before submitting:

What happened — describe the actual behavior you observed

Steps to reproduce — numbered steps someone else can follow to see the same bug

Expected behavior — what should have happened instead

Actual behavior — what actually happened (be specific)

Pod serial number — which pod is affected

Firmware version — check the control application or module info

Screenshots or video — attach visual evidence if applicable

Error messages — copy exact error text, do not paraphrase

When it started — first noticed date/time, and whether anything changed before it started

Frequency — happens every time? Intermittent? Only under certain conditions?

1. Create the Bug Report in ClickUp

Go to ClickUp > Standard Operating Procedures > Product & Technology (or designated bug tracking space — TBD, confirm with Sarid)

Create a new task with the title format: \[BUG\] \[Priority\] Brief description

Example: \[BUG\] \[Critical\] Pod 007 — all audio output dead after firmware update

Set the priority flag in ClickUp to match the severity level

Paste all gathered information from step 2 into the task description

Assign to Sarid (or designated developer — TBD)

Add the tag bug

1. Notify the Right People

\[table-embed:1:1 Priority | 1:2 Who to Notify | 1:3 How | 2:1 Critical | 2:2 Sarid + Tav + Justin | 2:3 ClickUp mention + Telegram message immediately | 3:1 High | 3:2 Sarid + Tav | 3:3 ClickUp mention | 4:1 Medium | 4:2 Sarid | 4:3 ClickUp assignment (no extra notification needed) | 5:1 Low | 5:2 Sarid | 5:3 ClickUp assignment (no extra notification needed) |\]

5\. Follow Up

Critical: Check for response within 4 hours. If no response, escalate to Tav and Justin directly

High: Check for response within 24 hours

Medium/Low: Verify the task is picked up in the next sprint planning

Cross-References

Product & Technology SOP 0 — Technology Stack Overview (to identify affected components)

Product & Technology SOP 1 — Firmware Release Process (bugs found after release)

Product & Technology SOP 4 — MQTT System Health Check (diagnostic tool for many bugs)

Manufacturing SOP 05 — Quality Control (bugs found during QC)

TBD Items

\[table-embed:1:1 Item | 1:2 Who Needs to Confirm | 2:1 Designated bug tracking space/list in ClickUp (separate from SOPs?) | 2:2 Sarid | 3:1 Default assignee for bugs (Sarid? Fahmy? Rotating?) | 3:2 Sarid | 4:1 Bug triage meeting — does one exist? Frequency? | 4:2 Sarid |\]

Done When

Bug report contains ALL required information (checklist in step 2 fully completed)

Task created in ClickUp with correct priority, assignment, and tag

Appropriate people notified per the notification table

Bug is acknowledged by the assignee within the SLA window

Can't determine the priority: Default to High and let the developer downgrade if appropriate. Never default to Low for an unknown issue

Can't reproduce the bug: Still report it — note "intermittent, could not reproduce on demand" and include all context you have

No response within SLA: Escalate one level up (Medium -> Tav, High -> Tav + Justin, Critical -> Justin directly)

Bug reappears after being marked fixed: Reopen the original task, do NOT create a new one. Add a comment with the new occurrence details

Not sure if it's a bug or a feature request: File it as a bug. The developer will reclassify if needed

SOP 3: Pod Software Update — On-Site

Purpose

Update pod software during a scheduled service visit, ensuring the update is applied safely with proper verification and fallback options.

Frequency

As needed — during scheduled maintenance visits or when a critical update requires on-site deployment.

Owner

Fahmy, Tav

Maintainer

Wina

Time Needed

1-2 hours (including travel prep, update, and verification)

Steps

Phase 1: Pre-Visit Preparation (Before Leaving)

Confirm current pod firmware version — check the firmware version tracker or contact the pod owner

Download the update package from MinIO:

Connect to MinIO via VPN

Download the correct firmware binary for the target version

Verify checksum matches the release record

Download a backup of the current configuration from the pod (via VPN if possible)

Prepare the update toolkit:

Laptop with update software installed

USB cable (for manual flash if needed)

Network cable (if WiFi is unreliable at site)

Previous firmware version binary (for rollback)

Printed/saved copy of this SOP

Confirm the visit with the pod owner/site contact — agree on time window, ensure pod is accessible

Check Manufacturing SOP 07 (UPS System) — do NOT perform updates if there are known power issues at the site

Phase 2: On-Site Update

Connect to the pod's local network — use Wireguard VPN or direct network connection

Verify connectivity to all 9 ESP32 modules — each module should be responding to MQTT heartbeats

If any module is unresponsive BEFORE update: document which module(s) and report separately. Do NOT proceed with update until all modules are online (unless instructed otherwise by Sarid)

Create a restore point — back up current firmware and configuration from the pod control PC

Apply the update:

Run the update script/tool from the control PC

Monitor progress — do not interrupt the process

If updating individual ESP32 modules, update one at a time and verify each before proceeding to the next

Wait for all modules to restart — allow up to 2 minutes per module for restart and reconnection

Phase 3: Post-Update Verification

Verify all 9 ESP32 modules are online — check MQTT heartbeats (see Product & Technology SOP 4)

Verify firmware versions — confirm each module reports the new version number

Run abbreviated QC check — based on Manufacturing SOP 05 (Quality Control), verify:

All lighting zones responding to commands

Audio output functioning on all channels

Sensor data being received correctly

Session can be started and stopped

No error messages on the control PC

Test a full session cycle — run a short test session (2-3 minutes) to verify end-to-end functionality

Document the update — record in the firmware version tracker:

Pod serial number

Previous version -> new version

Date and time of update

Any issues encountered

QC check results (pass/fail per item)

Phase 4: Handoff

Inform the pod owner/operator that the update is complete

Brief them on any changes they might notice (new features, changed behaviors)

Provide your contact details for the next 24 hours in case issues arise

Update the ClickUp task — mark the update as complete, attach documentation

Cross-References

Product & Technology SOP 0 — Technology Stack Overview (understand the system before updating)

Product & Technology SOP 1 — Firmware Release Process (the release process that creates the update package)

Product & Technology SOP 2 — Software Bug Reporting (if issues found during/after update)

Product & Technology SOP 4 — MQTT System Health Check (post-update verification)

Manufacturing SOP 04 — ESP32 Flash & Firmware Loading (manual flash procedure if update script fails)

Manufacturing SOP 05 — Quality Control & Testing (full QC checklist for reference)

Manufacturing SOP 07 — UPS System (power stability check before updating)

TBD Items

\[table-embed:1:1 Item | 1:2 Who Needs to Confirm | 2:1 Update script/tool name and location | 2:2 Sarid | 3:1 Can updates be done module-by-module or must they be simultaneous? | 3:2 Sarid | 4:1 Minimum battery/UPS charge level required before updating | 4:2 Fahmy | 5:1 Standard toolkit checklist — anything else needed? | 5:2 Fahmy |\]

Done When

All 9 ESP32 modules running the target firmware version

Abbreviated QC check passed (all items green)

Test session completed successfully

Firmware version tracker updated

Pod owner informed and briefed

ClickUp task updated with results

Module unresponsive before update: Do NOT update that module. Document and report. Update remaining modules only if Sarid approves

Update fails mid-process: Do not retry immediately. Check error logs. If the module is bricked, use manual flash (Manufacturing SOP 04)

Module unresponsive after update: Wait 5 minutes for full restart. If still unresponsive, attempt manual flash of previous firmware version

Multiple modules failing after update: STOP. Roll back ALL modules to previous version. Report to Sarid immediately

Power outage during update: Wait for power to return. Check UPS status. Verify which modules completed the update and which did not. Resume or roll back as appropriate

QC check fails after update: Document which checks failed. If Critical (pod non-functional), roll back. If non-critical, report to Sarid for guidance

Cannot connect to pod network: Check Wireguard VPN configuration. Try direct network cable. If still failing, contact Sarid for network troubleshooting

SOP 4: MQTT System Health Check

Purpose

Verify that the pod's MQTT messaging system is functioning correctly — all modules are communicating, messages are being delivered, and latency is within acceptable limits.

Frequency

After every firmware update (mandatory)

During scheduled maintenance visits

When any pod communication issue is reported

Weekly automated check (TBD — confirm with Sarid if automated monitoring exists)

Owner

Sarid (CTO), Fahmy

Maintainer

Wina

Time Needed

15-30 minutes per pod.

Steps

1. Connect to the MQTT Broker

Establish VPN connection to the pod network (Wireguard)

Connect to VerneMQ broker using an MQTT client:

Broker address: TBD (confirm with Sarid)

Port: TBD (confirm with Sarid — typically 1883 for unencrypted, 8883 for TLS)

Credentials: TBD (confirm with Sarid)

Recommended client: MQTT Explorer, mosquitto\_sub, or the control application's built-in tools

1. Verify Module Heartbeats

Subscribe to the heartbeat topic for all 9 modules:

Topic pattern: TBD (confirm with Sarid — e.g., pod/{serial}/module/+/heartbeat)

Wait 60 seconds and verify that ALL 9 modules have published at least one heartbeat

Record which modules responded:

\[table-embed:1:1 Module # | 1:2 Status | 1:3 Last Heartbeat | 1:4 Notes | 2:1 1 | 2:2 | 2:3 | 2:4 | 3:1 2 | 3:2 | 3:3 | 3:4 | 4:1 3 | 4:2 | 4:3 | 4:4 | 5:1 4 | 5:2 | 5:3 | 5:4 | 6:1 5 | 6:2 | 6:3 | 6:4 | 7:1 6 | 7:2 | 7:3 | 7:4 | 8:1 7 | 8:2 | 8:3 | 8:4 | 9:1 8 | 9:2 | 9:3 | 9:4 | 10:1 9 | 10:2 | 10:3 | 10:4 |\]

3\. Check Message Latency

Send a test command to each module via MQTT (e.g., status request)

Measure round-trip time — from command sent to response received

Acceptable latency: TBD (confirm with Sarid — typically <100ms on local network)

Flag any module with latency >500ms as potentially problematic

1. Verify Message Delivery

Send a test command and verify the module executes it (e.g., toggle a light, play a test tone)

Verify the response is received on the expected topic

Check for message loss — send 10 rapid messages, verify all 10 are received and acknowledged

1. Check Broker Health

Check VerneMQ status:

Connected clients count (should be at least 10: 9 modules + control PC)

Message queue depth (should be near zero — high queue = delivery problems)

Uptime (unexpected restart indicates issues)

Check broker logs for errors or warnings (log location TBD — confirm with Sarid)

1. Document Results

Record the health check results in the ClickUp task or maintenance log:

Date/time of check

Pod serial number

All 9 module statuses

Latency measurements

Broker health status

Pass/Fail overall assessment

Any issues found and actions taken

Common Issues and Troubleshooting

\[table-embed:1:1 Issue | 1:2 Possible Cause | 1:3 Action | 2:1 Module not publishing heartbeats | 2:2 Module powered off, firmware crash, WiFi disconnected | 2:3 Check module power LED. Reboot module. Check WiFi signal strength | 3:1 High latency (>500ms) | 3:2 Network congestion, WiFi interference, broker overloaded | 3:3 Check network traffic. Move module closer to access point. Restart broker | 4:1 Broker connection lost | 4:2 Broker crashed, network down, VPN disconnected | 4:3 Check VerneMQ process. Check network connectivity. Reconnect VPN | 5:1 Messages not being delivered | 5:2 Topic mismatch, ACL permissions, queue overflow | 5:3 Verify topic names match. Check VerneMQ ACL config. Clear message queue | 6:1 Intermittent disconnections | 6:2 WiFi signal weak, power fluctuations, firmware bug | 6:3 Check signal strength. Check UPS (Manufacturing SOP 07). Report to Sarid (SOP 2) |\]

Cross-References

Product & Technology SOP 0 — Technology Stack Overview (MQTT architecture context)

Product & Technology SOP 1 — Firmware Release Process (health check required post-release)

Product & Technology SOP 2 — Software Bug Reporting (report issues found during health check)

Product & Technology SOP 3 — Pod Software Update On-Site (health check is part of post-update verification)

Manufacturing SOP 05 — Quality Control & Testing (MQTT verification is part of QC)

TBD Items

\[table-embed:1:1 Item | 1:2 Who Needs to Confirm | 2:1 VerneMQ broker address and port | 2:2 Sarid | 3:1 MQTT credentials (username/password or certificate) | 3:2 Sarid | 4:1 Heartbeat topic pattern | 4:2 Sarid | 5:1 Acceptable latency threshold | 5:2 Sarid | 6:1 Automated monitoring — does it exist? What does it check? | 6:2 Sarid | 7:1 VerneMQ log file location | 7:2 Sarid | 8:1 MQTT client tool recommendation for technicians | 8:2 Sarid |\]

Done When

VPN connection established to pod network

All 9 modules publishing heartbeats (or non-responding modules documented)

Message latency within acceptable limits for all responding modules

Test commands executed and responses verified

Broker health checked (clients, queue depth, uptime)

Results documented in ClickUp or maintenance log

Any issues found have been reported via SOP 2 (Bug Reporting)

Cannot connect to VPN: Check Wireguard config. Verify VPN server is running. Contact Sarid

Broker appears down: Check if VerneMQ process is running on the server. Restart if needed (with Sarid's approval). This is Critical priority — file immediately via SOP 2

More than 2 modules unresponsive: This is a High priority issue. Do NOT attempt to fix individually — report to Sarid for coordinated diagnosis

All modules unresponsive: Check if the control PC is running and connected. The issue may be at the PC level, not the modules. This is Critical — notify Sarid + Tav immediately

Health check results inconsistent (pass, then fail, then pass): This indicates an intermittent issue. Run the check 3 times over 15 minutes and document all results. Report as Medium priority