AI is watching this Manhattan doorbell camera, counting the people walking by. It counts what they're wearing and if they have an umbrella. Sometimes the best forecast is just paying attention. How this works → | ||
| Counted 0 people in last 2 hours | ||
 | 0% | wearing long sleeves |
 | 0% | wearing pants |
 | 0% | using umbrellas |
AI is watching this Manhattan doorbell camera, counting the people walking by. It counts what they're wearing and if they have an umbrella. Sometimes the best forecast is just paying attention.
A simple person-detection AI model called YOLO ("You Only Look Once") watches the video feed continously, and draws a box around every pedestrian it sees. As people walk through the frame, it assigns each one a unique ID. It can follow people across multiple video frames, even if they briefly disappear behind someone else.
For each person, the system saves a single clear image to analyze what they're wearing.
All those "best shot" frames get sent in a batch to a LLM, specifically Google's Gemini, a multimodal AI that can interpret images. It looks at each person and answers a few simple questions: shorts or pants? Short sleeves or long sleeves? Carrying an umbrella?
The answers get tallied across everyone who walked by. The crowd is producing a weather report for you.
Thanks to Brian for technical pointers, and James for the idea to do clothes in addition to umbrellas!