Peridot is a cutting-edge AR mobile pet-sim game played via camera passthrough. Our primary goal was to use AR to make the pet feel like a real creature in the physical world.

To do so, we used AI to build a 3D mesh of the world in realtime via a depth estimation algorithm. As a Unity engineer on the team, I was responsible for making it so the pet used this 3D data to do things like hop up on your table, be occluded when it walks behind a wall, and pathfind realistically around your environment.

Another major gameplay feature I helped with was the object recognition gameplay. When you point your phone camera at the ocean, a semantic segmentation model detects it and has your Peridot bring you back a kelp gift. When your phone is viewing a person, the Peridot can run up to them and ask for a scratch behind the ears.

Outside of the AR features, the game had to... be a game! I implemented features like friending, gifting, questing, notifications, streaks, news, and telemetry. I think the strongest engineers are those who understand the entire system they're working with, and to that end I took on server, tooling, and tech art work where I could.