A post about using AI for RSS reader summeries caught my attention this morning… (Linked)…Basically the OP was concerned about giving personal data to AI companies… My responses - Here’s a developer’s perspective. When I started building Muuted ( an rss reader), I wanted it to be as private as possible. I legit struggled with whether or not to add APIs for the summary feature, and I have gone back and forth on it. I landed on a compromise that might set your mind at ease about using LLM cloud APIs.
At first Muuted’s summaries ran on a hosted endpoint i was gettign fo free from my Github pro sub serving a small OpenAI model. It worked, but as you pointed out, it went to OpenAI servers, and I read through their privacy policies and I wasn’t comfortable with it. I’m a privacy nut so it bugged me. So I moved the whole thing onto a locally hosted 4070Ti. It worked and i used it for about three months, but one issue… I had to build in redundancy if my local server went down. So my only options at the time were to spend a fortune a hosted GPU or be backed up by a hosted model only if my local server became unreachable . I settled on the latter (first that same OpenAI endpoint, then Groq after the free one shut down). If people are paying for a feature I have to be able to deliver it. So we’re right back to square one.
As far as quality… The local model was ok but not great. The app was also growing and taxing my 4070 so I tried a few larger models. The GPU could not handle them alongside everything else I had going on, so I had to drop summaries back to the smaller model and the results went back to mediocre. I sat with that for a few weeks but it was really annoying me… so I did some research and landed on Anthropic. Here’s why… Anthropic’s commercial terms say they do not train on API traffic, and on top of that, I built the code so nothing about the reader goes with the request: no account, no email, no reading history. Each story is summarized once and cached for everyone on Muuted to share. I ran a week long side-by-side comparison with Anthropic’s Haiku. It wasn’t even close and I made the decision to flip it… keep Haiku live and use my own local model as the fallback. And like I said… I am a privacy nut and I was comfortable with it.
So I guess what I’m getting at here is privacy can be built into almost anything. The dev just has to want that. You’re likely not going to find that with anything that is ads based, so I would def lean towards subscription based apps or freeware, and make sure you read their privacy policies, because they should spell out if your data is being exposed to third parties.
If your bar is “nothing leaves my machine”… it can be done, but like you already pointed out, it will be expensive (needs a beefy GPU) and time consuming. Hope this helps set your mind at ease about LLM summeries in some way…