• 11 posts
  • 3 comments
Joined 7 years ago
Cake day: April 17th, 2019

What is Lemmy?

Lemmy is a self-hosted social link aggregation and discussion platform. It is completely free and open, and not controlled by any company. This means that there is no advertising, tracking, or secret algorithms. Content is organized into communities, so it is easy to subscribe to topics that you are interested in, and ignore others. Voting is used to bring the most interesting items to the top.

Changes

This release significantly reduces memory usage for the Lemmy backend. Metrics on production instances show a reduction up to 10 times. Here you can see the statistics from a few different instances. Read on for a technical explanation below. Note that the change only affects x86, there is no difference on ARM (e.g. Raspberry Pi), see here for details.

lemmy.ml (overall RAM usage including OS, Docker, PostgreSQL etc):

leminal.space (backend container only)

lemmy.world (backend API containers)

lemmy.world (backend federation and scheduled tasks)

How was such a major improvement possible? Memory usage in languages like Rust or C is managed by a so-called memory allocator. It requests large chunks of RAM from the operating system, and provides smaller chunks when needed in the program. For example, every string (such as post titles or markdown text) require chunks of memory to store them. When processing the data is completed, the memory should be freed and released, or reused.

Until version 0.19.19 Lemmy used the mimalloc memory allocator, which is supposedly better than the default glibc allocator. Yet a recent blog post points out that mimalloc does not play well with the tokio async runtime. It suggests to use jemalloc instead. Changing memory allocators is very simple in Rust, it only requires a single line of code. So we tried that, deployed the change on lemmy.ml and it immediately proved effective.

Upgrade instructions

There are no breaking changes with this release.

Follow the upgrade instructions for ansible or docker.

If you need help with the upgrade, you can ask in our support forum or on the Matrix Chat.

Thanks to everyone

We’d like to thank our many contributors and users of Lemmy for coding, translating, testing, and helping find and fix bugs. We’re glad many people find it useful and enjoyable enough to contribute.

Support development

We (@dessalines and @nutomic) have been working full-time on Lemmy for over five years. This is largely thanks to support from NLnet foundation, as well as donations from individual users.

If you like using Lemmy, and want to make sure that we will always be available to work full time building it, consider donating to support its development. A recurring donation is the best way to ensure that open-source software like Lemmy can stay independent and alive, and helps us grow our little developer co-op to support more full-time developers.

  • Any negative speech: especially harsh, slanderous or divisive, that provokes equally charged emotional responses.

    One other thing I’d like to add: It’s not just that rage-inducing content is out there, that’s always been there. It’s that it gets rewarded by big tech, by making sure that highly responded content, gets pushed to the top.

    Capitalist tech companies have an incentive to platform this content because it increases the on-site time, and their own popularity, to the detriment of their users mental health. It also helps them push pro-capitalist narratives that atomize and isolate people who should be united by a shared humanity and class status.

    tbh we’re somewhat guilty of this in lemmy also, with the Active sort allowing new comments to bump posts to the top of feeds.

    The only real way to fight it, is to keep an eye out for these toxic / combative threads and posters, and remove them if necessary. Since we don’t capitalist incentives to keep people on site or serve ads, it serves no benefit to keep rage-bait posters around.

The charts above show growth of active users on thelemmy.club (left) and lemmus.org which are suggested on join-lemmy.org. It shows that improvements to the official project site really have a (small) influence on user growth. There are also a few new features:

This past month we’ve been working on lemmy-ui bug fixes and visual improvements, and a few remaining lemmy back end issues to get ready for the 1.0 release. We’ve closed over 20 lemmy-ui milestone issues, and have only a few remaining. The only major issue left is performance-testing, to ensure all our database changes and optimizations are working correctly.

You can follow our progress with these milestone links:

If you’d like to see the new features and visual updates in production, you can visit voyager.lemmy.ml which deploys updates every night. Please help by testing the new features and reporting any problems.

If you have any experience with web development or want to learn it, consider contributing to lemmy-ui. It is written in standard Typescript with Bootstrap.

If you have experience with Kotlin or Android development, you can help contribute to lemmy’s open source android app, Jerboa.

Thank you to everyone who has helped out with testing, development, spreading the word about lemmy, and building communities. Your help has brought lemmy from an idea to one of the most vital pieces of software in the fediverse.

Here are the major changes from February:

Full list of changes by user

malsadev

iByteABit256

flamingos-cant

dessalines

Nutomic

Or see the full list of changes at the links below:


An open source project the size of Lemmy needs constant work to manage the project, implement new features and fix bugs. Dessalines and Nutomic work full-time on these tasks and more. As there is no advertising or tracking, all of our work is funded through donations. Even so there is barely enough time in the day, and no time for a second job. The only available option are user donations.

To keep it viable donations need to reach a minimum of 5000€ per month, resulting in a modest salary of 2500€ per developer. If that goal is reached we can stop worrying about money, and fully focus on improving the software for the benefit of all users and instances. We especially rely on recurring donations to secure the long-term development and make Lemmy the best it can be.

Donate

What is Lemmy?

Lemmy is a self-hosted social link aggregation and discussion platform. It is completely free and open, and not controlled by any company. This means that there is no advertising, tracking, or secret algorithms. Content is organized into communities, so it is easy to subscribe to topics that you are interested in, and ignore others. Voting is used to bring the most interesting items to the top.

Changes

This version fixes a handful of bugs. Most importantly it prevents a background task from blocking the Lemmy server entirely. Edit marks on comments are now hidden if it was edited within 5 minutes of creation. Also the browser cache usage is significantly reduced. Additionally ARM builds are fixed by using the default memory allocator on that platform.

Backend

  • Fixing active counts slow queries. by @dessalines in #5907
  • Only use mimalloc on x86 and purge images in background task by @Nutomic in #5893
  • Add missing comment depth check by @Nutomic in #5842
  • Dont sanitize RSS content manually (fixes #5850) by @Nutomic in #5852
  • Reduce false positives in URL blocklist to reduce scunthorpe problem by @Nothing4You in #5807

Frontend

  • Don’t show edit mark if comment was edited in less than 5 minutes by @jfaustino #3197
  • Increase bio max length to 1000 chars by @nutomic #3249
  • Change link from element.io to matrix.org by @nutomic #3250
  • Remove all caches (fixes #3195) by @Nutomic in #3248
  • Fixed ordering for search results by @Nutomic in #3219
  • Add search field to community sidebar by @Nutomic in #3217
  • Add checkbox for title only search by @Nutomic in #3220

Upgrade instructions

There are no breaking changes with this release.

Follow the upgrade instructions for ansible or docker.

If you need help with the upgrade, you can ask in our support forum or on the Matrix Chat.

Thanks to everyone

We’d like to thank our many contributors and users of Lemmy for coding, translating, testing, and helping find and fix bugs. We’re glad many people find it useful and enjoyable enough to contribute.

Support development

We (@dessalines and @nutomic) have been working full-time on Lemmy for over five years. This is largely thanks to support from NLnet foundation, as well as donations from individual users.

If you like using Lemmy, and want to make sure that we will always be available to work full time building it, consider donating to support its development. A recurring donation is the best way to ensure that open-source software like Lemmy can stay independent and alive, and helps us grow our little developer co-op to support more full-time developers.

What is Lemmy?

Lemmy is a self-hosted social link aggregation and discussion platform. It is completely free and open, and not controlled by any company. This means that there is no advertising, tracking, or secret algorithms. Content is organized into communities, so it is easy to subscribe to topics that you are interested in, and ignore others. Voting is used to bring the most interesting items to the top.

Changes

This release took a long time to complete due to a major performance problem which brought lemmy.ml to a crawl every time we tried to deploy the new version. It took a lot of testing (in production) to narrow it down to a single commit, and finally fix the problem.

The release itself contains numerous bug fixes and minor improvements:

Lemmy

Enhancements

  • Parallel federation sending by @phiresky in #4623
  • Reduce CPU usage for generating link previews by @phiresky in #4957
  • Switch from OpenSSL to rustls by @kwaa in #4901
  • Increase max post url length to 2000 characters by @dessalines in #4960
  • Increase max length of user bio to 1000 charactes by @dessalines #5014
  • Reduce maximum comment depth to 50 by @nutomic #5009
  • Resize post thumbnails by @nutomic #5107/files
  • Add category to RSS feeds by @nutomic #5030
  • Allow users to view their own removed/deleted communities by @dessalines in #4912
  • Add backend check to enforce hierarchy of admins and mods by @dessalines in #4860
  • Do pictrs transformations for proxied image urls by @dessalines in #4895
  • Enable more build optimizations by @nutomic in #5168
  • Calculate “controversial” ranking with exponent instead of multiply (just like Reddit) by @dullbananas in #4872
  • Automatically remove tracking parameters from URLs by @dessalines #5018
  • Relax timeout for sending activities by @Nothing4You in #4864

Bug Fixes

  • Fix admin notification for new user registration (fixes #4916) by @Nutomic in #4925
  • Allow community settings changes by remote mods @flamingo-cant-draw in #4937
  • Fix problem with connecting to Postgres with TLS @FenrirUnbound in #4910
  • Fix bug when commenting in local-only community by @dessalines in #4854 and @abdel-m in #4920
  • Fix scheduled task to delete users with denied applications by @Nothing4You in #4907

API

  • Return image dimensions and content type in API responses by @dessalines in #4704
  • Adding a show_read override to GetPosts. by @dessalines in #4846
  • Add show_nsfw override filter to GetPosts. by @dessalines in #4889
  • Require authentication for site metadata fetch endpoint by @dessalines in #4968
  • Add the ability to fetch a registration application by person_id by @dessalines in #4913
  • Order community posts by published data, not id by @dullbananas in #4859
  • Throw error when non-mod posts to mod-only comm or when URL is blocked by @flamingo-cant-draw in #4966
  • Add option to search exclusively by post title by Carlos-Cabello #5015

Database

  • Approve applications in transaction by @Nothing4You in #4970
  • Use trigger to generate apub URL in insert instead of update, and fix query planner options not being set when TLS is disabled by @dullbananas in #4797

Lemmy-UI

  • Fix full-size post images. by @dessalines in #2797
  • Fix modlog ID filtering. by @dessalines in #2795
  • Allow Arabic and Cyrillic characters when signing up or creating community by @SleeplessOne1917
  • UX - Swap “Select Language” and “Cancel/Preview/Reply” button locations around in commentsReverse order of buttons in Reply TextArea
  • Fix jump to content by @SleeplessOne1917
  • Fixing peertube and ordinary video embeds. by @dessalines in #2676
  • Changing sameSite cookie from Strict to Lax. by @dessalines in #2677
  • Remove show new post notifs setting. by @dessalines in #2675
  • Fix memory leak around emojis on server render by @makotech222 in #2674
  • Enable spellcheck for markdown text area by @SleeplessOne1917 in #2669
  • Pre release dep bump by @SleeplessOne1917 in #2661
  • Add ability to fill magnet link title on post creation. by @dessalines in #2654
  • Registration application view by @SleeplessOne1917 in #2651
  • Add torrent help by @dessalines in #2650
  • More moderation history by @dessalines in #2649
  • Fix tribute related bug by @SleeplessOne1917 in #2647
  • Remove min and max length from password input when using login form by @SleeplessOne1917 in #2643
  • Remove trending communities card from home. by @dessalines in #2639
  • Set data-bs-theme based on the presence of “dark” in theme name by @SleeplessOne1917 in #2638
  • Fixing modlog filtering to allow admins and mods to filter by mod. by @dessalines in #2629
  • Fix issue from logo bugfix by @SleeplessOne1917 in #2620
  • Make more post params cross-postable by @SleeplessOne1917 in #2621
  • Fix wonky comment action icon button alignment by @SleeplessOne1917 in #2622
  • Prevent broken logo from crashing site by @SleeplessOne1917 in #2619
  • Add rate limit info message. by @dessalines in #2563
  • Fix getQueryString by @matc-pub in #2558

New Contributors

  • @abdel-m made their first contribution in #4920
  • @johnspurlock made their first contribution in #4917
  • @FenrirUnbound made their first contribution in #4910
  • @kwaa made their first contribution in #4901
  • @Daniel15 made their first contribution in #4892

Full Changelog

Upgrade instructions

This upgrade could take as long as ~30 minutes for larger servers, due to needing to recalculate controversy ranks for all historical posts.

There are no breaking changes with this release.

Follow the upgrade instructions for ansible or docker.

If you need help with the upgrade, you can ask in our support forum or on the Matrix Chat.

Thanks to everyone

We’d like to thank our many contributors and users of Lemmy for coding, translating, testing, and helping find and fix bugs. We’re glad many people find it useful and enjoyable enough to contribute.

Special shout out to @SleeplessOne1917, @phiresky, @dullbananas, @mv-gh, @Nothing4u, @asonix, @sunaurus, @flamingo-cant-draw, and @Freakazoid182 for their many code contributions and helpful insights.

Support development

We (@dessalines and @nutomic) have been working full-time on Lemmy for over five years. This is largely thanks to support from NLnet foundation, as well as donations from individual users.

If you like using Lemmy, and want to make sure that we will always be available to work full time building it, consider donating to support its development. A recurring donation is the best way to ensure that open-source software like Lemmy can stay independent and alive, and helps us grow our little developer co-op to support more full-time developers.

What is Lemmy?

Lemmy is a self-hosted social link aggregation and discussion platform. It is completely free and open, and not controlled by any company. This means that there is no advertising, tracking, or secret algorithms. Content is organized into communities, so it is easy to subscribe to topics that you are interested in, and ignore others. Voting is used to bring the most interesting items to the top.

Major Changes

This v0.19.4 release is a big one, with > 200 pull requests merged since v0.19.3. As such we can only give a general overview of the major changes in this post, and without going into detail. For more information, read the full changelogs at the bottom of this post.

Local Only Communities

Communities have a new visibility setting, which can be either Public (current behaviour) or LocalOnly. The latter means that the community won’t federate, and can only be viewed by users who are logged in to the local instance. This can be useful for meta communities discussing moderation policies of the local instance, where outside users shouldn’t be able to participate. It is also a first step towards implementing private communities. Local only communities still need more testing and should be considered experimental for now.

Image Proxying

There is a new config option called image_mode which provides a way to proxy external image links through the local instance. This prevents deanonymization attacks where an attacker uploads an image to his own server, embeds it in a Lemmy post and watches the IPs which load the image.

Instead if image_mode is set to ProxyAllImages, image urls are rewritten to be proxied through /api/v3/image_proxy. This can also improve performance and avoid overloading other websites. The setting works by rewriting links in new posts, comments and other places when they are inserted in the database. This means the setting has no effect on posts created before the setting was activated. And after disabling the setting, existing images will continue to be proxied. It should also be considered experimental.

Many thanks to @asonix for adding this functionality to pict-rs v0.5.

Post hiding

You can now hide a post as a dropdown option, and there is a new toggle to filter hidden posts in lemmy-ui. Apps can use the new show_hidden field on GetPosts to enable this.

Moderation enhancements

With the URL blocklist admins can prevent users from linking to specific sites.

Admins and mods can now view the report history and moderation history for a given post or comment.

The functionality to resolve reports automatically when a post is removed was previously broken and is now fixed. Additionally, reports for already removed items are now ignored.

The site.content_warning setting lets admins show a message to users before rendering any content. If it is active, nsfw posts can be viewed without login.

Mods and admins can now comment in locked posts.

Mods and admins can also use external tools such as LemmyAutomod for more advanced tools.

Media

There is a new functionality for users to list all images they have previously uploaded, and delete them if desired. It also allows admins to view and delete images hosted on the local instance.

When uploading a new avatar or banner, the old one is automatically deleted.

Instance admins should also checkout lemmy-thumbnail-cleaner which can delete thumbnails for old posts, and free significant amounts of storage.

Federation

Lemmy can now federate with Wordpress, Discourse and NodeBB. So far there was only minor testing and these projects are still under heavy development. If you encounter any issues federating with these platforms, open an issue either in the Lemmy repo or in the respective project’s issue tracker. You can test it by fetching the following posts:

In order to improve interoperability with Mastodon and other microblogging platforms, Lemmy now automatically includes a hashtag with new posts. The hashtag is based on the community name, so posts to /c/lemmy will automatically have the hashtag #lemmy. This makes Lemmy posts much easier to discover.

Reliability and security of federation have been improved, and numerous bugs squashed. Signed fetch was broken and is fixed now.

Vote display user setting

There is now a user setting to change the way vote counts are displayed, called vote display mode.

You can specify which of the following vote data you’d like to see (or hide): Upvotes, Downvotes, Score, Upvote Percentage, or none of the above. The default (based on user feedback) is showing the upvotes + downvotes.

App developers will need to update their apps to support this setting.

RSS Feeds

RSS feeds now include post thumbnail and embedded images.

Security Audit

A security audit was recently performed on Lemmy. Big thanks to Radically Open Security for the generous funding, and to Sabrina Deibe and Joe Neeman for carrying out the audit. The focus was on federation logic, and discovered various problems in this area. Most of the problems are being mitigated as part of this release. Fortunately no critical security vulnerabilities were discovered.

This is already the third security audit of Lemmy, all organized by ROS. We’re greatly indebted to them for their support.

Other Changes

Full Changelog

Upgrade instructions

Warning: This version requires both a Postgres and Pictrs version upgrade, which requires manual intervention.

Follow the upgrade instructions for ansible or docker.

If you need help with the upgrade, you can ask in our support forum or on the Matrix Chat.

Thanks to everyone

We’d like to thank our many contributors and users of Lemmy for coding, translating, testing, and helping find and fix bugs. We’re glad many people find it useful and enjoyable enough to contribute.

Special thanks goes to Radically Open Security, @sleepless and @matc-pub for their work on lemmy-ui and lemmy-ui-leptos, @dullbananas for their help cleaning up the back-end, DB, and reviewing PRs, @phiresky for federation work, @MV-GH for their work on Jerboa and API suggestions, @asonix for developing pictrs, @ticoombs and @codyro for helping maintain lemmy-ansible, @kroese, @povoq, @flamingo-cant-draw, @aeharding, @Nothing4U, @db0, @MrKaplan, for helping with issues and troubleshooting, and too many more to count.

Support development

We (@dessalines and @nutomic) have been working full-time on Lemmy for over three years. This is largely thanks to support from NLnet foundation, as well as donations from individual users.

If you like using Lemmy, and want to make sure that we will always be available to work full time building it, consider donating to support its development. A recurring donation is the best way to ensure that open-source software like Lemmy can stay independent and alive, and helps us grow our little developer co-op to support more full-time developers.

This is a chance for any users, admins, or developers to ask anything they’d like to myself, @nutomic@lemmy.ml , SleeplessOne , or @phiresky@lemmy.world about Lemmy, its future, and wider issues about the social media landscape today.

NLNet Funding

First of all some good news: We are currently applying for new funding from NLnet and have reached the second round. If it gets approved then @phiresky@lemmy.world and SleeplessOne will work on the paid milestones, while @dessalines and @nutomic will keep being funded by direct user donations. This will increase the number of paid Lemmy developers to four and allow for faster development.

You can see a preliminary draft for the milestones. This can give you a general idea what the development priorities will be over the next year or so. However the exact details will almost certainly change until the application process is finalized.

Development Update

@ismailkarsli added a community statistic for number of local subscribers.

@jmcharter added a view for denied Registration Applications.

@dullbananas made various improvements to database code, like batching insertions for better performance, SQL comments and support for backwards pagination.

@SleeplessOne1917 made a change that besides admins also allows community moderators to see who voted on posts. Additionally he made improvements to the 2FA modal and made it more obvious when a community is locked.

@nutomic completed the implementation of local only communities, which don’t federate and can only be seen by authenticated users. Additionally he finished the image proxy feature, which user IPs being exposed to external servers via embedded images. Admin purges of content are now federated. He also made a change which reduces the problem of instances being marked as dead.

@dessalines has been adding moderation abilities to Jerboa, including bans, locks, removes, featured posts, and vote viewing.

In other news there will soon be a security audit of the Lemmy federation code, thanks to Radically Open Security and NLnet.

Support development

@dessalines and @nutomic are working full-time on Lemmy to integrate community contributions, fix bugs, optimize performance and much more. This work is funded exclusively through donations.

If you like using Lemmy, and want to make sure that we will always be available to work full time building it, consider donating to support its development. Recurring donations are ideal because they allow for long-term planning. But also one-time donations of any amount help us.

This is an opportunity for any users, server admins, or interested third parties to ask anything they’d like to @nutomic@lemmy.ml and I about Lemmy. This includes its development and future, as well as wider issues relevant to the social media landscape today.

Note: This will be the thread tmrw, so you can use this thread to ask and vote on questions beforehand.

Original Announcement thread

Rule #2 is possibly our most important one:

Be respectful. Everyone should feel welcome here.

Learn to disagree without being rude or disrespectful.

It can be difficult sometimes, since western social media thrives on collective outrage, and they knowingly ingrain this into us for years. But please do adhere to this rule, and it will make this place much more enjoyable.

We will not hesitate to issue temp bans (usually a day or two) for those who make everyone’s experience unpleasant.Hit the report button if you see this behavior.

Thanks!