What is it that makes a PC an AI PC? Beyond some vague hand-waving at the presence “neural processing units” and other features only available on the latest-and-greatest silicon, no-one has come up with a definition beyond an attempt to market some FOMO.
While Intel suggests that application developers will soon infuse all software with AI, and PCs must be ready for them, the workload that currently matters most is inferencing to power large language models.
Any AI PC worthy of the name must therefore perform inferencing quickly and well.
Inferencing is the process that transforms a submitted prompt into its response. Doing so needs a machine that can grind through multiple gigabytes of data – and even more matrix multiplications.
While such a compute-intensive task sounds like it could well stretch the capacities of a desktop PC (ChatGPT and its ilk run in colossal datacenters), I’ve discovered that a lot of the hardware we already have can do an adequate job of inferencing. Anyone can run a “good enough” chatbot on a PC – provided the PC has enough RAM and a mid-range GPU.
That means a GPU with at least 8GB of VRAM (and not one from Intel – at least at the moment, for lack of driver support). That’s not too pricey – which is good, because it’s table stakes. For RAM, you’d be happier with 32GB than 16GB, while 8GB simply won’t cut it. And that’s about it. That’s all you’re going to need.
Apple has finessed this exceptionally well – possibly entirely accidentally – with its M-series SoCs. Treating memory as “unified” – it’s all VRAM if it needs to be – all modern M-series Macs with at least 16GB of RAM can serve as an AI PC.
As it happens, four of the PCs I’ve purchased over the last decade meet these specs – including an eight year-old top-of-the-line monster purchased as a “VR PC” with an Intel 6700K CPU and then-top-of-the-line Nvidia GTX 980 Ti GPU. More or less in line with Moore’s Law, it’s somewhat slower than the newer machines in my collection, but it runs a chatbot just fine – locally.
While not purchased as AI PCs, my more recent acquisitions each inference at a speed comparable to the completions generated by OpenAI’s GPT-3.5-Turbo – virtually indistinguishable from their bigger cousin both in speed, and in the quality of the completions they generate.
To test these systems, I feed them a fairly sophisticated prompt, instructing them to act as an “agent” – solving a problem by generating JSON files that get fed into another program. (“Small pieces loosely joined” still applies in the Age of AI.) And it all … just works.
So it turns out I already have AI PCs all over the place. Who knew? What’s more, there are a lot of AI PCs out there. Tens of millions. Which is good – because we’re going to need them.
While Microsoft would have us pasting every sensitive document into Copilot for analysis – thereby giving Redmond datacenters a sniff of our most confidential workflows – plenty of documents are simply too sensitive to be shared at all.
What about the documents so hot they shouldn’t even live on a networked machine? What about medical information? Or classified documents? So much of what we work with comes with strings attached, preventing it from being shared freely.
Legal documents frequently fall into this category – the kinds of tasks where the recently released SaulML legal language model could be enormously helpful. If a lawyer or paralegal had that model running on their local PC, they could freely pump documents into it, ask it to generate a close reading, even new contract terms – all without worrying about any confidential information leaks. Many law firms would run toward that sort of tool, screaming “Shut up and take my client’s money!“
I prototyped that tool last weekend – cutting the full-fat open source SaulLM model down to size, so it could work on my fleet of AI PCs. I’m not a lawyer, so I can’t fully judge its quality. I did, however, paste into it vast sections of a contract from a prospective business partner, then ask it some hard questions – without any fears of leaks.
Even if the lawyers don’t love it, the rest of us – confronted with endless, abstruse terms and conditions that we ignore at our peril – will find it very useful.
Likewise the hardware vendors – who rely on you junking the PC you bought to work from home three years ago for their revenue cycle – won’t like to admit it, but you probably already have a perfectly serviceable AI PC. Just not the AI PC that vendorland wants you to rush out and buy. ®
