AI-driven 3D object descriptions — using Google Cloud Vision API and Shapes XR to make virtual reality environments more inclusive and understandable for all users.
The goal was to enhance accessibility within virtual reality environments by using AI-powered image recognition to generate real-time descriptions of 3D objects — making immersive spaces more inclusive and understandable for all users, including those who rely on audio or assistive descriptions to navigate digital spaces.
Before building anything, I tested four AI platforms to evaluate their ability to recognise and describe 3D objects within a VR context — each assessed on recognition accuracy, response clarity, relevance to 3D environments, and API integration capability.
Developed a working prototype in Shapes XR to simulate AI-assisted object description in a VR environment. The prototype had two core features:
Used PopAi to generate and format natural-language object descriptions from the Cloud Vision API output. Integrated those descriptions into Shapes XR with voice functionality — ensuring a smooth, non-intrusive experience that blends AI-driven assistance with immersive VR design.
The technical challenge was keeping the AI layer invisible to the user — the experience needed to feel like the VR environment was simply explaining itself, not like a separate tool had been bolted on.
This project explored the intersection of AI, XR, and accessibility. Through research, testing, and hands-on prototyping, it demonstrated how smart technology can make virtual environments more inclusive for everyone.
Key takeaway: Designing for accessibility in emerging tech isn't optional — it's essential. With the right tools and intent, we can make immersive spaces more human, one feature at a time.