Poor lemmy server went from constant 80-100% cpu load to gently coasting.

That thing must have been dragging things down for months, if not years.

Update: (Sunday 23:00) And we’re back. 2 million posts have been deleted, and their communities have been disabled in the bot. That should keep things going a little longer.

Update 2: (Monday 10:00) All the communities have been synchronized and are up to date again. That’s great news! Before the purge it would take at least 24 hours to handle all communities after an outage, and even then it’d be lagging behind.

The bot is currently paused on account of a huge cleanup taking place.

The amount of post has once again outgrown the size of the server, so a lot of communities and posts will be purged, focusing first and foremost on those with the least amount of interaction. The server had been running flawlessly for the past few months, but apparently we hit a bottleneck at around 7 million posts. Which is nothing to be sneered at.

Anyway, for the upcoming week or more, a huge cleanup action will take place (deleting nearly 2 million posts), during which the bot will not run. This should give some breathing space, and also slow down the growth going forward.

So for all the Lemmit users: sorry for the delay. Also sorry if your favorite community got wiped, and no, this is not negotiable.

Server space is getting crowded, so I’ll be deleting a lot of communities that have tons of posts, but very little followers. And when I say tons of posts, I mean about 660.000 - a little under 10% of the total Lemmit posts. To put it into perspective - the next biggest Lemmy instance, LemmyWorld, has about 550k posts. I’ll be deleting more posts than they have in total. Talk about a hobby getting out of hand…

Since the Lemmy server model was never designed for this use case (rightfully so), and I’m running this on a teensieweensie server - this is asking a lot of the database. Therefor the site will be blocked from regular traffic to lessen the load. Expect this maintenance time to last through the weekend, maybe longer.

With Vultr’s upcoming price hike, I’m planning to migrate this instance storage bucket to a more affordable alternative. Unfortunately, it’s rather large, (over 700 GB in thumbnails), so it’s going to take a while to transfer.

Long story short: the bot will be not be posting new content for a couple of days - probably about a week.

All migrations and updates have completed. The server is now all up to date again, and the bot is playing catch up.

The good news: The migration is complete, and I’ve even managed to update the version to 0.18.5 (was stuck on 0.18.4-beta8 for the longest time).

The sad news: Cloudflare is having some issues, so nobody is able to access the new server at this time. Oh well. It’ll probably be fixed Saturday morning, and I’ll turn the bot back on.

Migration complete, and the bot has caught up on the 24 hour gap that it was offline. It only took like 12 hours this time, while in the past it was closer to taking an entire day. It probably helped that the new VM is dual core, even though the bot itself only ever makes 1 request at a time, so I didn’t expect this much of an improvement.

Since its inception, the Lemmit instance has been controversial. That might be an understatement, but let’s roll with it for now. One of the major issues people have with the bot, is the cross-posting of “interactive” Reddit posts, ie posts where the value lies in interacting with the OP, like AskMen, AskWomen, and AmITheAsshole. Personally, I fully agree with that viewpoint, but I didn’t feel like interfering with supply and demand - in the sense that AmITheAsshole, for some reason, is the most subscribed community on this server.

That might change though. Earlier this week, I disabled the posibility to request new subreddits. This weekend I will follow that up by disabling the scraping of so-called interactive communities. So in order to facilitate that, I created a list of all the communities on this server (posted separately in !about@lemmit.online), and I will check each of them to see if they should be disabled. The goal is to keep a list of “content only” (or at least “content mostly”) communities, where the value lies in the link that’s provided or in the body of the self-post - not in the comment section. I’m sure this is going to be a disappointment to some people, but I do agree with the sentiment that this is better for Lemmy as a whole.

Edit: It is done. All 816 communities have been checked and and 110 of those have been purged from updates. I am sure some mistakes were made - that some communities have been disabled or left intact when they shouldn’t have. If that’s the case, reach out to me, and I’ll fix it.

Since it’s impossible to see all the communities on here without logging in (mandatory NSFW filter), and I’m the only one with an account on here, here’s a list of them. This list will be a snapshot, so the subscriber count will not be up to date, but I’m sure you’ll figure it out.

• !unixporn@lemmy.ml

In the short time since this instance and bot launched, I’ve been seeing the same questions resurface multiple times. This is totally understandable, since the concept of a Fediverse is still new to most (myself included), and this server is not like the others.

Q: What is Lemmit?

A: Lemmit is a Lemmy instance specifically designed for archiving Reddit content. Users can request new subreddits to be included in the archiving process by posting in the !requests@lemmit.online community. It is powered by an open source python bot, which periodically checks the request list, adds new requests to the queue, and continuously monitors the Hot feed of those subs for new posts to cross-post here.

Q: Does it synchronize comments?

A: No, that would be impossible. Considering there are thousands of posts already on Lemmit, many of them having at least several hundred comments on Reddit, often buried in deep layers, it simply wouldn’t be feasible to index those for more than a few posts, let alone keep them up to date.

Unfortunately, this means that archiving certain subreddits, such as Ask Historians/Men/Women/Hyperintelligentshadesofthecolourblue-type subs, is going to be rather pointless.

Q: Can it send comments back to Reddit?

A: No, it cannot. The purpose is to help bootstrap the Lemmy platform, not to serve as a bridge between the two networks. Also, see the answer about synchronizing comments.

Q: Can I request any subreddit?

A: Technically, yes. However, as the list of subs grows, the time it takes to update all of them will also increase. I do not have strict guidelines in place for this, so I’m relying on your common sense (hoooo boy). At some point, I will probably have to either stop accepting new requests or disable scraping for very low-traffic communities. Nope. Automatic requests for new subreddits are closed. You can send me a PM if you think a specific subreddit would be a valuable addition to the Lemmyverse, but you’ll have to make a strong case. Going forward, I only expect the amount of subreddits to go down, since I don’t intend to let the server grow endlessly large.

Q: Does this use the API? Will it keep working after July 1st?

A: Nope, it uses a combination of the public feed and scraping old.reddit.com. So, as long as those are still available, it will continue working. And even if they close those sources, there will probably be new ways to achieve the same effect. “Content, eh, finds a way.”

Q: This is spam, can you stop?

A: First of all, I apologise for the inconvenience. All you have to do is block @bot@lemmit.online, and none of its posts will ever show up on your instance. If you you don’t want anyone else on your server to be exposed to this bot/instance, you should convince your admin to defederate from lemmit.online. Since there are no other users on here, there will be no harm done.

Obviously I could stop, because running this server and software is only ever going to cost me time and money. But for the reasons listed above, I still think this server is a useful addition to the lemmyverse at this time. But I’m looking forward to the day where I can turn the bot off because it’s no longer needed.

Q: What started this?

A: Okay, nobody asked this, but I’m going to tell you anyway. After Reddit made it clear that they are effectively killing third-party apps and implementing plenty of other anti-end user decisions, I realized that I would either have to accept not being able to access my time-wasting content or have to do so in a rather uncomfortable way (either through the official app or old.reddit.com for as long as they’ll allow it to exist).

Being a stubborn developer, naturally, I chose option C: Have my own Reddit. With blackjack, and hookers. This way, I would still be able to access my beloved content without being beholden to Reddit’s mood swings and abusive relationship tendencies.

Besides that, I also know that Content is King. So I’m order to counter the network effect (No users because no content, No content because no users), I figured it would be better to have some inorganic content to bootstrap the adoption of Lemmy.

Q: Are NSFW subreddits allowed?

A: Absolutely. Like I said: Blackjack and hookers.

Q: My request isn’t picked up by the bot!

A: That isn’t a question. But yeah, the process isn’t flawless yet. I’m trying to iron out all the bugs as I encounter them. In the meantime, feel free to re-request the subreddit by making a second post. No harm done. New subreddit requests are closed for the time being.

Q: No new posts are showing up at all on Lemmit

A: If no posts are appearing on the Lemmit Frontpage (sorted by NEW), it’s possible that the bot has crashed or is stuck on something. Since no software is flawless, this sometimes happens. I usually fix this as soon as I’m aware, and I’m happy to say that these kinds of fatal errors are becoming less and less frequent. However, they may still occur, and as a human with needs of sleep and other responsibilities, I’m not always able to fix them immediately.

Q: Posts aren’t showing up on my instance, what’s up?

A: Due to the spammy nature of the bot, some server admins choose to block this server, and that is completely understandable. So first of all, make sure to check the instances link in the footer of your home server. If Lemmit is the Blocked Instances list, you’re out of luck.

When you have verified that Lemmit is not blocked on your instance, try unsubscribing, waiting a little, and then re-subscribing. That tends to fix things.