Since its inception, Let’s Encrypt has been sending expiration notification emails to subscribers that have provided an email address to us. We will be ending this service on June 4, 2025.
↫ Josh Aas on the Let’s Encrypt website
They’re ending the expiration notification service because it’s costly, adds a ton of complexity to their systems, and constitutes a privacy risk because of all the email addresses they have to keep on file just for this feature. Considering there are other services that offer this functionality, and the fact many people automate this already anyway, it makes sense to stop sending out emails.
Anyway, just a head’s up.
Thom Holwerda,
I think it’s a helpful feature to have. I have received some notifications already when there was an error and TBH I’m thankful they sent them instead of having to learn about it through downtime. Not having error notifications is a loss of a nice feature, but at the same time I recognize that using a free service means that users shouldn’t feel too entitled.
I think everyone automates lets encrypt because it comes this way out of the box. But even automated systems can break after updates and configuration changes. Operators won’t necessarily be aware of problems until certificates expire potentially months later. If error notifications were optional, I think many admins would still opt in to receive them.
I do not understand any operator that wouldn’t have a check against their tls cert. Thats a big first thing I do task. Heck I do it even for my stupid sites.
Bill Shooter of Bul,
I still think notifications are beneficial even if just to contact users about service changes. it can be frustrating when there is no notice.
Anyway, here’s some PHP code from a status page used to detect when a certificate is expiring. Others may find it helpful. Set verify peer appropriately.
My goodness it looks like it’s missing fclose, haha.
Whatever automated process you use should include a “send me an alert e-mail if this fails”. And if your automated process doesn’t support that, you should find a different process to use.
Shoot, we even have e-mail-to-SMS enabled for our Let’s Encrypt process. And we attempt renewal at least 2 weeks before the expiration date, just in case things fail. Even 2 weeks is a bit of a short cushion, but it works for us.
The nice thing about Let’s Encrypt is you control everything locally, so you can configure things to alert you any number of different ways. No need to rely on a remote 3rd party to send alerts.
phoenix,
Maybe I’m in the minority, but it seems totally reasonable for service providers to be able to send notifications. If you don’t want those notifications then fine, they can be opt-in, but | don’t see a good reason to declare other people shouldn’t have them. I work with some large vendor APIs and honestly I much appreciate it when they send out a notification that “yo, something’s changing” or “we’re switching over servers on this date” etc.
Granted letsencrypt certificated have a fairly large buffer before outages become production issues, but nevertheless in principal these notifications are a good thing and I don’t see a legitimate reason to argue against having them.
Of course, I appreciate that LE is a free service,
phoenix,
A bit of a tangent here, but I wanted to ask: how are you doing SMS?
Are you using these public email to sms gateways?
https://smsemailgateway.com/
I used to do this myself, automating processes and emailing alerts to my phone, but the alerts were extremely unreliable. Several of us raised this issue in tmobile support forums. They confirmed the delivery failures, and individually escalated all our cases, but practically every case had the same conclusion, they could not fix delivery issues due to their spam filtering solution. The only thing that could be done was “wait until it works again”. BTW I am not asserting everyone has these issues, but clearly there were many of us who did. We asked if they could whitelist messages from ourselves to ourselves, since we’re obviously not sending spam to our own phone numbers, but they said their system doesn’t support whitelisting. I’m not sure if any carriers do…?
For me personally, sometimes I’d get the SMS alerts from my own servers for weeks and then see them abruptly stop for several days without a recourse. It’s still something I’d find useful, but it was just too unreliable in practice.
Do you have experience with sms providers that are more reliable than the public email to SMS gateways that I linked to? Any suggestions?
I use Voodoo SMS (https://www.voodoosms.com/). My use is incredibly low, so I paid a few £ for some credits a few years ago and haven’t run out yet! Never had it fail. I use it to have my home server text me when it boots, so if there’s a power cut or kernel panic, I get a text rather than wondering why I haven’t had any emails for a while
markr,
Cool, I’d need to find something similar in the US.
Another idea I had was to create my own app that would have it’s own alerts that didn’t rely on SMS. It would have to rely on 24×7 IP traffic. In principal it should work but I’ve noticed a lot of android apps fail to stay connected in the background. Since I don’t do android dev I’m not sure how difficult it is to overcome android’s energy saving modes. And I’m not sure what the implications are for battery life and data plan. If it’s just 1% battery then I don’t care, but 10% would be a lot. I think android offers it’s own notification channels through google servers, but I don’t have any interest in going that route.
The benefit of SMS is that all phones do it out of the box. I may end up trying this again, only with a reliable provider next time.
Very useful note. Thank you, Thom.
I run my own nextcloud, email, hosting, yada, yada from my home office behind dual WAN. Since upgrading to dual WAN, I never managed to get Let’s Encrypt to automatically renew. I counted on the emails to quickly kill one of the WAN links to get the automatic renewal process to succeed.
So I should perhaps finally take the time to figure out a fix for my situation. =)
If disabling one of the WAN connections makes the renewal work, the quick and dirty solution would be to wrap the renewal script with `ip link set xyz down` before and `ip link set xyz up` after
markr,
I think most of my systems call the certbot directly without a script. Of course you can create your own script and wrap it, but it might be better to reinstall and do things right, haha.
I’m curious why WAN configuration would matter at all…it just needs access to the website. Oh I wonder if Shiunbird is running the certbot in standalone mode? If so I guess it could be binding to the wrong interface? Maybe in 50 years time Shiunbird’s kids will write up a nice article about how an obscure certificate authority daemon was failing and Thom Holwerda’s kids will post it to future osnews. And our kids will just randomly chat about it in the comment section
Oh yes, making sure it’s binding to the right interface is the correct solution
I had an issue recently where an nginx reverse proxy was connecting to the backend service from the wrong IP, turned out I’d got my NAT config in nftables wrong and it was getting NATted. So if something is binding to the wrong interface even though you’ve told it the correct interface, try checking your NAT config if you’re using that!
My setup is not ideal, and I don’t have time to improve it – I never started from zero since my first server when I was 14 (I am 38 now haha). On the bright side, I still haven’t lost any data. =)
One fixed IP and one PPPoE connection reach pfsense (snort, firewall, routing, yadayada).
There’s a CNAME for most things.
But a round-robin A record for email (MX does not like CNAME)
DNS is on an external provider, except reverse on my interfaces (for email) and that’s done by my carriers.
Behind that, there’s haproxy in TCP mode (SNI check) to send requests to correct servers. Everything now is across 3 FreeBSD servers EXCEPT email, and that is still a Linux VM running mailcow (I want to migrate to FreeBSD as well to also have a triple master setup).
So when I call from freebsd (or mailcow calls) certbot renew, sometimes the verification tries to go out through a different WAN (load balancing), and then the check fails. I think certbot has a flag to skip IP verification.
So my renew process is… CNAME pointing to single IP A instead of round robin, and for the MX entry, change it to single IP instead of round robin. Mark one gateway down on pfsense. Renew certs. Revert changes.
Yes, I could automate all of that (or finish my setup!), or perhaps terminate SSL at haproxy, or maybe have a jail or VM just to handle certs, but real life has been getting on the way. =(
Shiunbird,
I’m afraid I’d be more helpful if you were running linux, but on freebsd I’m out of my depth
Like markr said in another post, I have occasionally seen daemons using the wrong IP on the wrong interface and obviously that won’t work. Packet dumps usually make it clear that only one side of traffic is working.
I’d say start with some packet traces to diagnose the issue, once you know that you’ll be in a better place to make a fix.
I have a hypo on how to sort this put, but real life is getting on the way.
The whole manual renewal brouhaha takes me 3 minutes to do, and I need a day to test my idea. (((
I appreciate the suggestions, anyway!
No policy based routing so its trying to use the primary interface source address when replying to traffic received on the secondary link…
Normal clients don’t break because of happy eyeballs.
SCTP would be great in this scenario, but sadly it’s not widely supported.
Lets encrypt is awesome, anything they need to do to stay in business is much appreciated. I use them not because it saves me money, but because its easier and faster than anything else. Plus it kills the dumb argument people use to not use TLS because it cost $7 a year or something like that.
Bill Shooter of Bul,
I don’t recall prices ever being that cheap.
I appreciate lets encrypt as well. Although technically domain validated CAs wouldn’t really need to exist at all with better standards. Browsers work this way because they evolved around CAs that verified real world identities (ie green certificates). CAs aren’t necessarily to prove domain ownership though. We should be able to create our own certificates and put the certificate fingerprint in DNS to prove ownership.. Look at DKIM (the email standard) as an example of how to cryptographically prove ownership without a CA.
Alas, I know this is wishful thinking, we’ve made CAs integral even for things like domain validation where they aren’t needed.