What it is
YogaSense is a progressive web app (PWA) that watches a user's yoga pose through the webcam and gives spoken corrections. It also offers guided breathing, a timed meditation mode, saved routines, a leaderboard and an augmented reality (AR) garden scene. A team built it in 24 hours at the Err_404 6.0 hackathon.
The problem
The repository describes YogaSense as an app for real-time pose correction. The code addresses this by measuring joint angles from the camera feed and comparing them with target ranges for each pose. It then tells the user which joints to bend or extend.
How it works
The app is a Next.js 15 project (App Router) with a PostgreSQL database. Pose detection runs in the browser, and the code never uploads the video.
| Area | Where | What it does |
|---|---|---|
| Pose practice | src/app/(app)/poses/[name]/webcam-component.tsx | Runs MediaPipe on the camera stream, draws the skeleton and collects feedback. |
| Pose rules | src/lib/yoga-poses.ts | Defines eight poses as lists of joint-angle limits. |
| Spoken feedback | src/app/api/speech/route.ts | Turns feedback text into audio with ElevenLabs. |
| Routines | src/app/(app)/routines/ | Lets users chain poses with set durations and log each run. |
| Breathing and meditation | src/app/(app)/breathing/, meditation/ | Timed breathing patterns and a meditation timer with pose tracking. |
| Heart rate | src/app/api/heartrate/route.ts | Accepts heart-rate readings from an outside client. |
| AR scene | src/app/(app)/yoga-studio/scene.tsx | Shows a 3D garden over the rear camera with A-Frame. |
| Leaderboard | src/app/api/leaderboard/route.ts | Ranks users by breathing-session points. |
| Notifications | src/actions/fcm.ts, src/lib/firebase.ts | Stores push tokens and sends Firebase Cloud Messaging (FCM) alerts. |
Pose correction loop
- The browser loads the MediaPipe Pose Landmarker (the "heavy" model) and starts the camera.
- For each video frame, the landmarker returns 3D body landmarks.
getJointAnglecomputes the angle at a joint from three landmarks, such as shoulder, elbow and wrist.- A pose defines a minimum or maximum angle for each joint.
getJointFeedbackreturns a "bend" or "extend" message when an angle is out of range. - The component keeps the last 1.5 seconds of feedback. It reports a joint only if at least 90% of recent frames flag it, which filters out jitter.
- The message goes to
/api/speech, and the browser plays the returned audio. - When the timer ends, the app saves the run:
PoseDatafor the session and onePoseFeedbackrow per joint reading, with time and angle.
Data model
The Prisma schema in prisma/schema/ has these main models.
User,SessionandAccountcome from Better Auth.Routine,RoutinePoseandRoutineUsagedescribe a routine and each time it is run.PoseDataandPoseFeedbackhold per-pose results, so the app can chart angles over time.Breathingrecords session length and pattern, and feeds the leaderboard.HeartRateandFCMTokenstore watch readings and push tokens.
What I built
I built the webcam pose detection and feedback, the speech output, the saving of poses and routines with the performance views, and the heart-rate ingestion and display. Jay built the breathing exercises and logging, authentication and profiles, the leaderboard and chatbot, and Firebase notifications. Sundaram built the routines, meditation mode, landing page, theme and AR studio. A teammate (kryyo1441) wrote the pose constraints and corrections and added the pose images and 3D assets.
Design decisions
Angle rules instead of a trained pose classifier. Each pose is a short list of joint limits. This makes the feedback explainable ("extend your left knee") and easy to tune, and the limits differ per side in some poses, which suggests hand tuning. The cost is that every new pose needs hand-written limits, and the thresholds are fixed for all body types.
Run the vision model client-side. MediaPipe runs in the browser with the GPU delegate. This avoids sending video to a server and removes a network round trip from the feedback loop. The model files load from a CDN at runtime.
Smooth feedback before speaking. Speaking every frame-level correction would be noisy and would flood the text-to-speech API. The 90% agreement rule over 1.5 seconds, plus a guard that plays one clip at a time, keeps the voice usable.
Use a PWA. The app registers a service worker with Serwist and shows an install button. One codebase serves desktop and phone.
Push alerts as a social signal. When a user finishes a pose, the server sends a "Pose Completed" notification to every other user who granted permission. This appears intended to nudge other users to practice.
Known limits
These come from reading the code.
- The OmGuru chatbot in
src/components/chatbot.tsxis rule-based. It matches keywords such as "beginner" against fixed pose lists. It does not call a language model. - The AR scene shows a static 3D garden model. The repository has no animated instructor.
/api/heartrateexpects readings from an outside device, and the watch client is not in this repository. The route does not check the caller's identity.- The leaderboard counts breathing sessions only.
Run it locally
Requirements: Node.js, npm and a PostgreSQL database.
-
Install dependencies. The
postinstallscript runsprisma generate.npm install -
Create a
.envfile withDATABASE_URL,BETTER_AUTH_URL,GOOGLE_CLIENT_ID,GOOGLE_CLIENT_SECRETandELEVENLABS_API_KEY. AddFIREBASE_SERVICE_KEY(a service account JSON string) to enable push notifications. -
Create the tables. The repository has no migration files, so push the schema directly.
npx prisma db push -
Start the development server and open
http://localhost:3000.npm run dev
Reference
| Area | Choice | Used for |
|---|---|---|
| Framework | Next.js 15, React 19, TypeScript | Pages, server actions, API routes |
| Database | PostgreSQL with Prisma 6 | Users, routines, pose and breathing history |
| Auth | Better Auth (email and password, Google) | Sign-in and sessions |
| Vision | @mediapipe/tasks-vision | Pose landmarks in the browser |
| Voice | ElevenLabs (eleven_multilingual_v2) | Spoken corrections |
| AR and 3D | A-Frame, three.js | Garden scene |
| PWA | Serwist | Service worker and install prompt |
| Notifications | Firebase Cloud Messaging | Push alerts |
| UI | Tailwind CSS, shadcn/ui, Recharts, Framer Motion | Layout, charts, breathing animation |
| State | Zustand | Routine progress |
Outcome
YogaSense was built at Err_404 6.0, a 24-hour hackathon at M.H. Saboo Siddik College of Engineering in Navi Mumbai on 15 February 2025. The team had four members. The hackathon page records no placement.
The app is deployed at yogasense.vercel.app. The repository was created on 15 February 2025, and the last commit is dated 19 February 2025.