Dragging CUDA-Only AI onto a Mac Without Losing Your Mind

Most interesting ML code was written by someone who assumed, the way you assume gravity, that there was an NVIDIA GPU under the desk. The assumption is so total it’s invisible. It’s baked into the imports, the install instructions, the one line three files deep that says device="cuda" like it’s reading a law of physics aloud.

Then you try to run it on a Mac, and the law of physics files a complaint.

Here’s the good news, the thing nobody tells you because it’s not heroic enough to brag about: getting CUDA-first code onto an M-series Mac is not a research project. It’s the same handful of moves, over and over. Once you’ve done two of these ports you’ve basically done all of them, the way once you’ve assembled two pieces of flat-pack furniture you’ve made peace with the little hex key. So here are the moves in one place. The individual case studies - PULSE , bitsandbytes , and eventually AudioCraft - link back here instead of repeating them, because life is short and the hex key is always the same.

A note on authorship: I direct agents that write these patches, set the constraints, and judge the results. I know what “running” is supposed to look like; I am not the person typing the Python. Read this as a field guide to recognizing the six monsters, not as kernel-engineer war stories.

And full disclosure on the Python, because it’s worse than you think: Python is not my language. I’m a JavaScript/TypeScript person who spent fifteen years before that writing crummy WordPress plugins, and to this day I see a variable wearing a $ like a little hat and think, yeah, that’s fine, that’s reasonable. I can read Python the way you can read a menu in a country you’ve never visited - enough to point at what I want - but write it fluently, in a language where apparently no function has ever once volunteered the shape of its own arguments and a comment is treated like a war crime? No. That part’s the agent’s, all of it.

1. Device selection: stop hardcoding the church you pray to

The original code says .cuda() everywhere. On a Mac the GPU isn’t CUDA, it’s MPS - Apple’s Metal-backed PyTorch device. So the first move is to quit naming a specific god and just ask for the best one in the building.

The pattern is a tiny device.py that picks MPS → CUDA → CPU in that order, with an environment-variable escape hatch (SOMETHING_DEVICE=cpu) for when you need to force the slow-but-reliable path to debug. Then you thread that one DEVICE value through the whole codebase and exterminate every hardcoded .cuda() and device="cuda" you can find. It’s tedious, it’s mechanical, it’s the single most reliable hour you’ll spend - and it’s exactly the kind of search-and-replace toil an agent should be doing while you watch.

2. The MPS fallback: the seatbelt you set once and forget

Metal does not implement every operation PyTorch knows about. Sooner or later the model reaches for some niche op, finds a hole where Metal should be, and crashes.

The seatbelt is one environment variable: PYTORCH_ENABLE_MPS_FALLBACK=1. With it set, any op Metal can’t do quietly hands that one operation back to the CPU, runs it there, and returns to the GPU like nothing happened. You usually want this on. The catch is that a fallback op drags data across the CPU/GPU border every time it runs, and that border has a toll. If one unimplemented op sits inside your hottest loop, your “GPU port” can somehow end up slower than just running the whole thing on CPU - a machine paying tolls to commute to a job two desks over. So: set the fallback for safety, then, if it’s slow, hunt the op that’s tripping it and decide whether to pin it to CPU on purpose or rewrite around it.

3. Dtype landmines: the float64 that isn’t there

This is the crash you will hit first and curse most, so let’s defuse it now: MPS does not support float64. Not slowly, not with a warning - at all. Some library casually does a double-precision calculation that nobody on a CUDA box ever noticed, MPS hits it, and the whole run face-plants with an error that does not, of course, say “I don’t do float64.” It says something cryptic three abstraction layers away.

The fix is almost always “find the spot promoting to float64 and tell it float32 is fine,” because for inference it virtually always is. While you’re in there, know that the two half-precision formats behave differently on Apple silicon too - bfloat16 and float16 don’t have identical support or numerical behavior, and precision bugs love to hide in that gap. If the model’s output goes subtly insane rather than crashing outright, suspect a dtype. (If “dtype” is a fog, the tensor entry is the five-minute version: it’s just what flavor of number is in the box.)

4. Dependency archaeology: the packages that only speak CUDA

Some dependencies began life as love letters to NVIDIA. The first question used to be whether they would install on a Mac at all. Now it is often more complicated: which backend installs, which features actually work, and whether any of them reach the Apple GPU.

bitsandbytes is the useful example. Current releases can install on macOS through a CPU backend, but “runs on a Mac” and “has a native, useful Metal backend” are not the same claim. I learned how wide that gap is by trying to build one.

For an old repo, the move is still to make CUDA-shaped dependencies such as xformers, bitsandbytes, and triton optional - wrap the imports so their absence is a shrug, not a death, and route to whatever non-CUDA path exists (often plain PyTorch attention instead of xformers, which is slower but real). Tangled up with this is straight-up archaeology: these repos are frequently pinned to a five- or six-year-old Python and a PyTorch from a previous geological era, and you have to drag the pins forward to something that runs on current Apple silicon without nudging the model into behaving differently. And don’t forget the deps that aren’t pip at all - ffmpeg, dlib, and friends live at the OS level and have to be installed the boring way (Homebrew) before any of the Python works.

5. Dead weights: the download links rotted years ago

Here’s a failure that has nothing to do with chips and everything to do with time: the model weights are gone. The README points at a Google Drive folder, a university server, an S3 bucket - and it 404s, because that link was somebody’s grad-school account and they graduated.

Always load weights from a local file first and download only as a fallback, the opposite of how these repos usually ship. Then go find where the canonical weights actually live now: they’ve almost always migrated to HuggingFace, or to the maintaining library’s own host (dlib’s model files, for instance). Half of “this old model doesn’t work anymore” is not a code problem at all. It’s a missing-persons case.

6. Sanity-checking the port: prove it, don’t vibe it

The most dangerous state is “it ran without crashing,” because that feels like success and isn’t. MPS will happily produce numbers that are quietly wrong - a dtype quirk, a fallback op returning something subtly off - and a model emitting confident garbage looks identical to a working one until you check.

So check. Start with the plain CPU path as an oracle. Compare the smallest operation that can expose a meaningful disagreement, using bit-exact checks for packed values and explicit tolerances for floating point. Then benchmark separately. Parity is evidence about correctness; timing is evidence about speed. The bitsandbytes port found a real optimizer bug this way before the first Metal kernel was written.

Then set realistic expectations: a Mac’s unified memory is generous, but parameter count still decides what’s usable. A model that’s a breeze on a 40GB datacenter card might run on your laptop the way a tour bus runs down a bike path, technically forward motion, deeply unadvisable. Knowing which size is actually usable on your Mac is the last move, and it’s the one that turns “I got it running” into “I actually use this.”

That’s the whole hex key. Six monsters: a hardcoded device, a missing op, a forbidden number, a CUDA-only dependency, a dead download, and a port that lies about working. Every one of these ports is just those six in a different costume. Recognize the costume and the fight gets boring - which, for this kind of work, is exactly the goal.

Two case studies are now live. PULSE is what happens when all six monsters arrive in a six-year-old trench coat. The bitsandbytes experiment begins with a CUDA-shaped dependency and ends with the correctness harness finding a bug in backends the Mac could not even run. AudioCraft is still on the way. Same six monsters, different victim.