Hello all!
I have been solidly ill for about 2 months, but I can finally see the light at the end of the tunnel. To that end, I am able to actually write some stuff this month, instead of last month’s phone-it-in excuse of an update. And what a month its been!
To add to that, next month (June 10th) is Lemmy.zip’s 3rd Birthday! If Lemmy.zip were a small child, it would apparently be able to ride a tricycle. How about that eh. Anywho, keep an eye out for our birthday celebrations :)
Server Updates
Status Page Updates
First, we have a new status page coming! It may already be in place by the time you read this. Currently we’ve been using Betterstack for the status page, but it has a limited number of sites we can monitor. We could use UptimeKuma, but I’ve never liked the style of it. Anyway we’ve swapped now, and the status page will be hosted at status.dotzip.ltd but all the sites we run will have status subdomains redirecting here, i.e. status.lemmy.zip will redirect there.
All our services can be documented on that page now, for a full picture of our infrastructure.
The Button
Season 3 of The Button starts very soon - 20:00 UTC on Monday 4th May (thats like, 2 days away!)
Originally it was just meant to be an April Fools joke but due to popular demand it keeps coming back.
More info is in the community :)
Piefed.zip has been relatively stable for the past month. That’s how I like my services :)
Let’s start with the performance issues. It kind of ties up everything that’s been happening this month on lemmy.zip nicely.
A little while ago, Lemmy version 0.19.16 was released. This introduced some database pooling/timeout changes, and as many of these things can do, this had the knock on effect of uncovering a previous bug with nested comments, which had managed to hide away.
Ultimately, deeply nested comment threads would cause lemmy to recursively process the whole comment chain until this caused a timeout in the software and a stack overflow of that process thread, which would lead all the threads becoming overwhelmed and unable to keep up with requests, causing lemmy to eventually restart. It becomes a bit of a federation death spiral:
- An instance sends deeply nested comments
- Lemmy recursively processes it
- Workers get tied up
- Queue fills
- New tasks can’t start
- Eventually → crash
This became apparent with the log line:
Timeout occurred while waiting for a slot to become available
Unfortunately, no amount of config tweaking could resolve this one.
For example, the above chart shows a 2 day period in early april - each drop is a restart of the server container as it gets timed out by a comment thread. Each time a new comment was posted, it caused the whole thread to be reprocessed and a new crash. This was exacerbated by the posting habits of some users on certain communities posting lots of deeply nested comments very quickly which then caused a loop of crashing and restarting -> next nested comment is processed -> crash and restart and so on. This is why I made the decision to temporarily defederate hexbear, as we’d gone about an hour of trying to process one set of comments in a thread over on Hexbear and it was causing about 10 seconds of uptime -> crash -> restart and so on.
Much more information on the error is available in that linked thread, and the it contains links to the github issue if you’re interested in seeing more in depth log errors.
I think there is still more work for the devs to do on the database pooling/timeout issues. This was ultimately two issues interacting to cause the restarts. Deeply nested comments eating up the CPU being recursively processed (fixed), and the added timeout functionality causing this to compound and crash the server.
Moving on to something a bit better, if you’re a moderator then you may be interested to know that our ModTools site for Lemmy.zip now allows you to link your own bot account, so instead of all actions coming from ZippyBot, you can use your own community bot!
Just make sure it’s set as a moderator in the community. I’ve been trialing this for a little while in the gaming community with @GamingBot@lemmy.zip.
ModTools is under heavy development, so please do expect bugs and things to not work. I also appreciate feedback on UI/UX and any missing functionality you’d like to see.
Please Don’t Be a Lurker!
If you’re new here - WELCOME! I hope you’re enjoying your time here :)
Just one small teeny-tiny request. The greatest gift you can give Lemmy.zip and Piefed.zip isn’t money, praise, or a signed copy of your self-published autobiography… maybe. Its participation.
Upvote the things you like. Start a discussion, debate, or ponder about how many monkeys it would require on how many typewriters to recreate this post in a feasible amount of time. Make new communities if you don’t see one that fits your oddly specific niche obsession.
(We don’t judge. Well, we try not to judge anyway.)
The fediverse naturally ebbs and flows, tides of people come and go. But if you’ve found yourself oddly attached to this strange little corner of the internet? Wonderful. Help it breathe. Help it grow. Help it be just a tiny bit weirder in the best possible way.
So if you’ve gone to the effort of clicking Sign Up and proving you’re not a robot (unless you are, in which case hello and welcome to our new AI overlords), then please, I beg of you:
Stick around. Add your voice. It really does make this place better.
Donations
Want to support us? We have a range of donation options to suit you!
Liberpay (Card, Paypal)
Liberpay is our newest donation method. You can donate with Paypal or with a card. The link is https://liberapay.com/dotzip
OpenCollective (Card)
Our classic donation option, but only supports card payments. The link is https://opencollective.com/lemmyzip
If you’re enjoying Lemmy.zip or Piefed.zip, please check out the OpenCollective page, we have a selection of one-off or recurring donation options. All funds go directly to hosting the sites and keeping the virtual lights on.
Ko-Fi (Card, Paypal)
We also have our Ko-Fi page if you’d rather use this site, which also supports Paypal and Card payments.
Thank you!
We continue to have some really kind and generous donators and I can’t express my thanks enough.
You can see all the kind donators in the Thank You thread - you could get your name in there too! (I am a little behind with this sorry but I will update it soon!)
Please form an orderly queue, there’s enough graphs for everyone!
Piefed.zip
For Piefed, we only store data for 7 days rather than 30 (because historically we had less storage space on the server) so all graphs are 7 days unless otherwise stated.
CPU:
RAM:
Current performance:
Image Storage:
Cloudflare Overview: (yes, they’ve changed the style of this graph)
Requests:
Bandwidth:
Visitors:
Traffic:
Lemmy.zip
CPU:
RAM:
You may noticed semi-regular restarts indicated here too - under investigation!
Disk space:
Current server performance:
Image storage:
Cloudflare Overview:
Requests:
Bandwidth:
Unique visitors:
Traffic:
And thats it for another month.
I do want to share the sad news that Lemmy.zip lost a member of the communtity recently. @GeekFTW@lemmy.zip and I first got chatting around 2 years ago when he was looking to move the away from it’s then-home due to performance issues, and it eventually settled on Lemmy.zip. Since then, I had got used to seeing his name pop up frequently in the local feed. The absence of that name a little while ago was sad, but it was recently confirmed that he had suddenly and unexpectedly passed away.
Lemmy is still a small community, and there are certain usernames you come to recognise and enjoy seeing in your feed. Sadly, we’ve lost one of those names, and I do hope you’ll take a minute to pause and remember the person behind the username. Whether you interacted with him regularly or just recognised his posts in passing, he was part of what made this community feel alive.
Online spaces can sometimes feel transient, but the people in them and the connections they create are very real.
Our thoughts are with his friends, family, and everyone here who felt his presence. He’ll be missed.
Thanks,
Demigodrick
