Greg Clarke

I’ve got 128GB RAM + 24GB VRAM on a 4090. I’ve managed to get a 400B parameter model running on a single board computer with 64GB RAM by using MMAP. But I want to run Kimi K3 locally so I would need a lot more RAM / bandwidth

Especially giving the lack of trust organizations rightfully have in big AI companies

My current workflow is passing a human written spec to an agent to implement with strict coding guidelines, architectural decisions, etc. The agent isn’t making any decisions about the abstractions to use, it’s just creating the objects and test suites. So I don’t mind the slower bandwidth because I’m running the heavy agentic lifting over night with no need for human supervision.

But I fully appreciate that my workflow isn’t the norm. In fact my workflow it’s the exact opposite the AI grifters like Sam Altman are selling because it still involves a human with knowledge of the systems making different decisions.

I just wish I could buy enough memory to run one of these models locally. Specially Kimi K3