Spectar · HoloLens · 2019–2023
Spectar put the building model on the jobsite at full scale, on a Microsoft HoloLens. I spent four years there figuring out what works when the screen is strapped to your face and the user is wearing a hard hat.
Role
Lead product designer
Timeline
2019 to 2023
Team
Two designers, a small engineering team and sales
Platforms
HoloLens 1 and HoloLens 2
Recognition
When I joined Spectar in 2019, the team was chasing a simple question: what if a crew could see the building model on the jobsite, full size, right where it was going to be built? I led the product’s design from there, on a headset almost none of our users had ever put on.
The problem
Buildings get designed in detailed 3D models and built from 2D drawings. Mistakes hide in that gap. A duct that runs into a beam that’s already on site. A wall that’s a few inches off. They tend to surface mid-install, when the fix means tearing out work that’s already done.
What made it hard
Mixed reality had very few design patterns yet, and none for construction. The HoloLens ran on a mobile chip, washed out in sunlight and wasn’t made to fit under a hard hat. And I didn’t know much about construction.
How I went at it
Spend a lot of time on jobsites. Get designs into the headset as early as possible. Keep at it until a foreman couldn’t imagine laying out a wall or checking an install without it. Over four years that meant two headsets, a lot of alignment experiments and one slightly ridiculous glove test.
2019 · HoloLens 1
Learning a new medium
On site with a Martin Bros. crew, 2019
The Projects menu on HoloLens 1
Logging in meant typing on a keyboard floating in midair
Gaze, air tap and a lot of patience
The first HoloLens gave you a handful of inputs: gaze to point, air tap to click, a bloom gesture for the menu, tap and hold to drag, and your voice. That was about it.
Voice helped, but jobsites are loud. We tried to stick with Microsoft’s “if you can see it, say it” voice guidelines, and made sure there was always another way to do it.
Even logging in was a design problem. Try typing a password on a keyboard that’s floating in the air.
Designing for all this had its own problem. The tools hadn’t caught up yet.
Process
Prototyping without the tools
Concept: building the model up from the floor plan
Menus and the placement tool, tested on my living room floor
The login screen, pinned up in my living room
Screenshots, in the headset
There weren’t real tools for designing in 3D space, so I started in Sketch and Photoshop, mocking up UI on top of photos from real jobsites.
Flat mockups lie, though. So I’d load those images onto the HoloLens and pin them around the room. You find out fast what’s too small, too dim or too far away.
Later, GLB models and ShapesXR let me test whole flows at real scale before engineering built anything.
The prototypes kept pointing at the same problem. None of it matters if the model is in the wrong spot.
Alignment
Making the model line up
Explaining drift: the farther you get from the anchor, the more the model slips
The later nudge tool, down on the slab
A QR code on a steel beam, found and aligned
A few inches off and nobody trusts it
Our first approach used a single holo target set on a known point, laid out by hand or with a total station. You placed the model on it, rotated it to face project north, and a nudge tool cleaned up the rest. It worked, but it took time and a steady hand.
We tried QR codes early on but dropped them over accuracy and UX. Microsoft’s HoloLens team talked us back into them for HoloLens 2, and this time they stuck (more on that below). The surprise was that people trusted them more, even when the accuracy was about the same. With a QR code the system does the aligning, not your hands. Less fiddly work made it feel easier and more trustworthy.
QR estimates were touchy about rotation, and codes spread farther apart did better. A short lock-in step, with an animation that showed it working, made setup a little slower and a lot more accurate. Worth it.
Getting it right in a test space was one thing. Then we took it outside.
Field research
Surviving the jobsite
The glove test
HoloLens 2 on a Spectar hard hat
Loading the model in tiles to keep the frame rate up
Gloves, glare and 60 frames a second
Hand tracking works great until you put on work gloves, and nobody on a jobsite takes their gloves off. So I bought a dozen pairs of the gloves crews actually wear and tested every one. Unsurprisingly, bright colors tracked best, though some shades worked a lot better than others.
Holograms wash out in sunlight, so we kept a tinted visor in our back pocket for the jobs that needed it. Hard hats were their own problem. Trimble sold a separate HoloLens 2 built into one, which meant buying a whole new headset. I mounted a regular HoloLens 2 onto a standard hard hat instead.
And the models were huge. Drop below 60fps and holograms start to swim and people get queasy. Early on we split models by level, zone and trade. Eventually Jesse, our chief engineer, built a rendering pipeline that did that splitting for you, so you could load an entire trade for a level at once.
Then HoloLens 2 came out, and we got to start over.
HoloLens 2
Starting over on new hardware
Concept: an eye-tracked hand menu
The Tools menu: visibility, benchmark and positioning
Download a project’s zones before you head out, then work without signal
Rebuilt from scratch
HoloLens 2 was lighter, better balanced and could track your hands. Instead of porting the old app, we rebuilt it. I led the screens, flows and hierarchy, and a second designer took the lead on color and type.
Menus moved onto your hand, so they went where you went. Everything lived in three sections, Projects, Tools and Settings, and each one opened the panel you needed for the task. Gestures picked up shortcuts, too. Tap and hold on the ground anywhere on site and the nudge tool showed up, so you could fix your alignment on the spot. The list of voice commands grew a lot, and you could lock your hands when you needed them for real work (more on that below).
Offline mattered just as much. Jobsites rarely have good internet, so the new app worked offline first: download a project before you go, and it synced whenever it found a connection.
Favorite features
Four features I still love
01 · Lockable hands
Turn your hands off
The problem
HoloLens is supposed to be hands-free, so you should be able to use your hands while you wear it. The catch: grab a drill or pull a tape and the headset might read it as a gesture and start tapping things you never meant to touch.
What we did
After a few rounds of ideas, we landed on a lock that lives on the back of your hand. Need to do some real work? Turn your hand away and slide the lock to switch your hands off. Ready to get back in? Turn it over and slide it back. Think slide to unlock on a phone, or the safety on a nail gun. This one shipped, and it felt really good to use.
02 · BIM Track
Drop a pin anywhere
The problem
Field issues usually get tagged to a floor, a room or a sheet number. Then somebody in the office has to figure out where exactly “by the east stair” is.
What we did
Our BIM Track integration let you drop a pin at a precise point, in the model or right there in the real world, and share that exact spot with the office. Back then, nobody else in AR was doing that.
03 · QR codes
QR codes, take two
The problem
We tried QR codes on HoloLens 1 and walked away over accuracy and UX. Then Microsoft’s HoloLens team pointed us back at them. HoloLens 2 is always scanning for codes at the system level, so reading them cost our app nothing extra, and they quickly became the no-brainer way to get localized on site.
What we did
I went deep on how QR codes work: formats, size, error correction, even which materials and angles the headset reads best. We landed on a simplified mobile QR format you could scan from a comfortable distance, paired with multi-point alignment for accuracy. Scanning a code skipped the project, level and model picker and dropped you right in. We also prototyped a one-point alignment that got the model up fast, then nudged you to scan a second code to tighten it up.
The edge cases
Codes we didn’t recognize, codes you didn’t have permission to use, codes with no point attached, no internet. Each one got a clear state, so nobody was left staring at a QR code wondering what happened.
Every path a scanned code could take, from “unsupported” to “active”
04 · Lenses (never built)
Pull down a new way of seeing
The idea
A full BIM model is a lot to look at, and what you need to see changes from task to task. So what if you could pull a lens down over your view, anytime? X-ray to see through walls. Highlighted edges so things are easier to read. Or just the backing in your framing model when that’s what you’re installing. Night vision goggles, but for BIM.
Why it didn’t ship
The gesture felt great. But we had more basic problems to solve first, and the cool interactions had to wait their turn. I still think it would’ve been neat.
How it turned out
Great for some jobs. Not for every day.
We plotted every use case by what it was worth against how accurate it had to be. A QC walk is useful even if the model is a few inches off. Laying out a door frame needs about an eighth of an inch.
Part of our use case board for wall and ceiling crews: every task by construction phase, with the accuracy it needed. Click to see the whole board.
Spectar was great at the first kind of job: QC walks, curved walls, custom fabrication, anything that’s hard to read on a 2D drawing. It showed a lot of promise on everyday install and layout work like backing, door frames and hanging pipe, but it never got widely adopted there. Setup time, a heavy headset and that accuracy bar kept it from becoming something crews reached for every day.
Spectar won a Red Dot Award for design concept in 2022. When Red Dot asked me what success would look like, I said it was “when construction workers insist on using our product to aid them in doing their work.” We got there with some teams. Not enough of them.
What I took away
Prototype where the work happens. A monitor lies about mixed reality, and an office lies about a jobsite.
People trust it more when the system does the work. The QR codes taught me that. Same accuracy, but nobody had to line anything up by hand.
The hardware sets the ceiling. Good design can’t make up for a device people don’t want to wear all day. I still think AR belongs on the jobsite. It’s the same idea I picked back up at Sitelink, this time on the phone already in everyone’s pocket.
From the archive
Four years of screenshots
Shoots, captures, specs, and a few ideas that never left Figma. Click any of them to see it bigger.