YogaSense

A yoga PWA that checks poses from webcam joint angles, speaks corrections, and tracks routines, breathing sessions and a leaderboard.

Err_404 6.0

🎓 M.H. Saboo Siddik College of Engineering

📌 Navi Mumbai, Maharashtra

🚀 24 hours Hackathon Project

What it is

YogaSense is a progressive web app (PWA) that watches a user's yoga pose through the webcam and gives spoken corrections. It also offers guided breathing, a timed meditation mode, saved routines, a leaderboard and an augmented reality (AR) garden scene. A team built it in 24 hours at the Err_404 6.0 hackathon.

The problem

The repository describes YogaSense as an app for real-time pose correction. The code addresses this by measuring joint angles from the camera feed and comparing them with target ranges for each pose. It then tells the user which joints to bend or extend.

How it works

The app is a Next.js 15 project (App Router) with a PostgreSQL database. Pose detection runs in the browser, and the code never uploads the video.

AreaWhereWhat it does
Pose practicesrc/app/(app)/poses/[name]/webcam-component.tsxRuns MediaPipe on the camera stream, draws the skeleton and collects feedback.
Pose rulessrc/lib/yoga-poses.tsDefines eight poses as lists of joint-angle limits.
Spoken feedbacksrc/app/api/speech/route.tsTurns feedback text into audio with ElevenLabs.
Routinessrc/app/(app)/routines/Lets users chain poses with set durations and log each run.
Breathing and meditationsrc/app/(app)/breathing/, meditation/Timed breathing patterns and a meditation timer with pose tracking.
Heart ratesrc/app/api/heartrate/route.tsAccepts heart-rate readings from an outside client.
AR scenesrc/app/(app)/yoga-studio/scene.tsxShows a 3D garden over the rear camera with A-Frame.
Leaderboardsrc/app/api/leaderboard/route.tsRanks users by breathing-session points.
Notificationssrc/actions/fcm.ts, src/lib/firebase.tsStores push tokens and sends Firebase Cloud Messaging (FCM) alerts.

Pose correction loop

  1. The browser loads the MediaPipe Pose Landmarker (the "heavy" model) and starts the camera.
  2. For each video frame, the landmarker returns 3D body landmarks.
  3. getJointAngle computes the angle at a joint from three landmarks, such as shoulder, elbow and wrist.
  4. A pose defines a minimum or maximum angle for each joint. getJointFeedback returns a "bend" or "extend" message when an angle is out of range.
  5. The component keeps the last 1.5 seconds of feedback. It reports a joint only if at least 90% of recent frames flag it, which filters out jitter.
  6. The message goes to /api/speech, and the browser plays the returned audio.
  7. When the timer ends, the app saves the run: PoseData for the session and one PoseFeedback row per joint reading, with time and angle.

Data model

The Prisma schema in prisma/schema/ has these main models.

  • User, Session and Account come from Better Auth.
  • Routine, RoutinePose and RoutineUsage describe a routine and each time it is run.
  • PoseData and PoseFeedback hold per-pose results, so the app can chart angles over time.
  • Breathing records session length and pattern, and feeds the leaderboard.
  • HeartRate and FCMToken store watch readings and push tokens.

What I built

I built the webcam pose detection and feedback, the speech output, the saving of poses and routines with the performance views, and the heart-rate ingestion and display. Jay built the breathing exercises and logging, authentication and profiles, the leaderboard and chatbot, and Firebase notifications. Sundaram built the routines, meditation mode, landing page, theme and AR studio. A teammate (kryyo1441) wrote the pose constraints and corrections and added the pose images and 3D assets.

Design decisions

Angle rules instead of a trained pose classifier. Each pose is a short list of joint limits. This makes the feedback explainable ("extend your left knee") and easy to tune, and the limits differ per side in some poses, which suggests hand tuning. The cost is that every new pose needs hand-written limits, and the thresholds are fixed for all body types.

Run the vision model client-side. MediaPipe runs in the browser with the GPU delegate. This avoids sending video to a server and removes a network round trip from the feedback loop. The model files load from a CDN at runtime.

Smooth feedback before speaking. Speaking every frame-level correction would be noisy and would flood the text-to-speech API. The 90% agreement rule over 1.5 seconds, plus a guard that plays one clip at a time, keeps the voice usable.

Use a PWA. The app registers a service worker with Serwist and shows an install button. One codebase serves desktop and phone.

Push alerts as a social signal. When a user finishes a pose, the server sends a "Pose Completed" notification to every other user who granted permission. This appears intended to nudge other users to practice.

Known limits

These come from reading the code.

  • The OmGuru chatbot in src/components/chatbot.tsx is rule-based. It matches keywords such as "beginner" against fixed pose lists. It does not call a language model.
  • The AR scene shows a static 3D garden model. The repository has no animated instructor.
  • /api/heartrate expects readings from an outside device, and the watch client is not in this repository. The route does not check the caller's identity.
  • The leaderboard counts breathing sessions only.

Run it locally

Requirements: Node.js, npm and a PostgreSQL database.

  1. Install dependencies. The postinstall script runs prisma generate.

    npm install
    
  2. Create a .env file with DATABASE_URL, BETTER_AUTH_URL, GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET and ELEVENLABS_API_KEY. Add FIREBASE_SERVICE_KEY (a service account JSON string) to enable push notifications.

  3. Create the tables. The repository has no migration files, so push the schema directly.

    npx prisma db push
    
  4. Start the development server and open http://localhost:3000.

    npm run dev
    

Reference

AreaChoiceUsed for
FrameworkNext.js 15, React 19, TypeScriptPages, server actions, API routes
DatabasePostgreSQL with Prisma 6Users, routines, pose and breathing history
AuthBetter Auth (email and password, Google)Sign-in and sessions
Vision@mediapipe/tasks-visionPose landmarks in the browser
VoiceElevenLabs (eleven_multilingual_v2)Spoken corrections
AR and 3DA-Frame, three.jsGarden scene
PWASerwistService worker and install prompt
NotificationsFirebase Cloud MessagingPush alerts
UITailwind CSS, shadcn/ui, Recharts, Framer MotionLayout, charts, breathing animation
StateZustandRoutine progress

Outcome

YogaSense was built at Err_404 6.0, a 24-hour hackathon at M.H. Saboo Siddik College of Engineering in Navi Mumbai on 15 February 2025. The team had four members. The hackathon page records no placement.

The app is deployed at yogasense.vercel.app. The repository was created on 15 February 2025, and the last commit is dated 19 February 2025.

Liked YogaSense? Here's my background.

Full-stack TypeScript, shipped to real users.