Mozilla and Google have extended their current search deal for another three years, multiple sources have told ZDNet.
The new search deal will ensure Google remains the default search engine provider inside the Firefox browser until 2023 at an estimated price tag of around $400 million to $450 million per year.
Mozilla officials are expected to announce the search deal’s extension later this fall, in November, when the organization is scheduled to disclose its 2019 financial figures.
This is definitely good news in terms of Firefox’ continued existence, but the uncertainty every time the deal is about to run out illustrates once again that this situation is entirely untenable. I have no idea how to solve this problem, but my wildest idea is large open source projects like major Linux distributions, GNOME, KDE, and so on, taking an active interest in investing manpower and other resources into ensuring Firefox remains the independent browser for Linux and BSD users.
God forbid I have to use Chrome.
This has been tried before. KHTML. Didn’t work. Turns out browser engines need a lot of work to maintain. About $400m per year apparently. I am not sure there are that many pure community projects with that sort of volunteer manpower.
Maybe community led development just doesn’t work for browsers anymore.
mkone,
I actually think that mozilla’s operation is extremely bloated. You really wouldn’t need more than a dozen developers full time to maintain a browser.. Expensive yes, but nowhere near $400m/yr. I believe a lot of that money is actually paying for mozilla’s admin costs and other projects. Unfortunately while much of that work has merit, there seems to be a problem finding ways to pay the bills without going back to google every time. 🙁
Also, since people were talking about mozilla’s new VPN service last time, I just happened across this article about VPNs logging data even though they had pledged not to.
https://www.pcmag.com/news/7-vpn-services-found-recording-user-logs-despite-no-log-pledge
EVERY VPN service logs data. Mozilla weren’t the first to be caught logging stuff, and they won’t be the last.
A VPN isn’t secure, and people shouldn’t assume it’s secure either. It’s about as secure as Tor is anonymous. All a VPN is is another layer of obscurity. With enough time, patience, and most of all, money, any VPN can be turned into a bona fide data mine. Just this time you’re paying for the privilege of being snooped on.
Never trust a 3rd party with your data. If you’re that paranoid about getting caught doing naughty stuff, you probably know much more secure methods of browsing the internet than just using a 3rd party paid VPN service.
But UK and US VPNs are required to log data by law. Which is why I wouldn’t trust Mozilla VPN.
Firefox codebase is ~25 million lines of code. I’d like to meet this 12 person team capable of handling that. Modern browsers are large, complex beasts. The fact that besides Mozilla only two companies have their own in-house browser engine should tell you just how hard it is, especially considering that those two companies – Apple and Google – are among the largest in the world.
Yeah, and even Apple had to use a 3rd party browser engine for their WebKit.
Mozilla is an extremely complex piece of software, just compile it from source to get the hint. I don’t know if the previous poster has worked on a large software base, but a dozen developers would most likely be overwhelmed trying to reign in and maintain mozilla.
javiercero1,
I didn’t say a dozen developers would be able to “maintain mozilla”, but they probably could maintain firefox if they needed to. And yes, I have worked on large code bases 🙂
Mozilla is Firefox.
Firefox is 20+ million lines of code and over 5000 contributors. The repository alone is several gigabytes.
I don’t think you have worked at a project remotely close to that scale if you keep reiterating than a dozen programmers can maintain such a massive and complex codebase. Just the coding diversity (5+ languages) would mean just about every programmer in that diminutive team would be proficient in C++/Rust/Javascript/Python. Good luck with that.
javiercero1,
Firefox is the browser, Mozilla are the developers. Go to Help | About to read it yourself.
And? As I said in another post, mozilla didn’t write all of that code and moreover the vast majority of it doesn’t change between releases.
You are too focused on the size of the code. The truth is that the size of the code isn’t the problem, more important is the rate of change. You might have to throttle back more ambitious changes, but once again I said maintain, not reinvent.
Additionally teams in larger corporations can actually suffer from diminishing returns even to the point where they actually become negative (ie adding more workers decreases productivity!).
https://en.wikipedia.org/wiki/The_Mythical_Man-Month
Many of the greatest innovations happen on small teams, meanwhile big companies are often plagued by overhead caused by “too many chiefs in the kitchen” making it very difficult to get things done (you’ve experienced this, haven’t you?). Of course they have way more resources too, which affects the balance. But I still maintain that you are underestimating what small teams can do.
I think a small team could have those bases covered 🙂
I’m sorry its just absurd to think that 12 devs could maintain it. That’s just a a fantasy. The best 12 could do would be to cause the ship to sink at a lower rate than just letting it go. Which honestly, after terminating the staff they did, might just happen anyways regardless of how many they put on it.
Bill Shooter of Bul,
That’s your opinion, I don’t think it’s a stretch at all for them to be able to “maintain it” indefinitely. The bigger question is how much they’d be able to improve it. And that’s open to debate, Keep in mind that firefox isn’t written from scratch but takes a lot technology from other open source projects. Also a lot of the subsystems are already abstracted, which greatly simplifies supporting new platforms and hardware. This is very common in large projects. When you’ve got A->B->C, it becomes immensely easier to update backend components without having to rewrite everything else. For example a person could add a new codec without too much effort by taking advantage of open source code and connecting it to an existing interface. Of course it takes some work, but not infeasible on the scale of a “google summer of code”. A full time team of 12 is like 48 google summers of code and that could be conservative because the fulltime devs are more experienced than interns. Like I said, you’d have to focus on things that are most important, but it is not unrealistic or impossible like some have suggested. There is a lot that can be done!
Alfman
I have to assume your connection with the realities of very large agile code bases is purely theoretical.
In most industry projects I’ve products I’ve worked. Management, maintenance, and support teams tend to be significantly larger than research and core development groups.
A group of a dozen developers would be tiny for maintaining even a medium software project with little visibility. Much less for something as massive, visible, and with such a diversity of contributions as Firefox.
Maintenance is not trivial at all. E.g. in commercial projects like Windows bug hunting and feature validation take far more effort in men hours than core and feature development.
javiercero1,
Erhum ad hominem cough. I am a very competent software engineer in the real world. Solving problems that give others difficulty is one of my strengths, believe it or not.
A lot of people are overwhelmed by a lot of things. A lot of developers wouldn’t be able to work on web browsers, design their own kernels or build their own linux distros, graphics engines, etc… but some people are up to the challenge. In painting everyone with the same brush, you get a picture of mediocre averageness rather than what a small talented group with a vision can actually accomplish.
Everywhere you look there’s complexity, but skilled people can tackle far bigger problems than you give them credit for.
It depends, if you’re conducting a complete code audit, that is extremely labor intensive since every line of code has to be reviewed. You’d need a ridiculous number of eyes to do that. But if you are tackling one bug or feature at a time, it is absolutely possible.
I’ve been in your chair before, convincing myself that something is not possible rather than thinking about ways that it might be done. Actually I’m usually the pessimistic one in conversations, it is a really nice change of pace being on the other side, thank you for this. Perhaps you can give me more opportunities to argue on the side of optimism in the future! I have to warn you though, being the pessimistic one takes a toll on your soul.
Alfman
This has nothing to do with optimism vs. pessimism. Perhaps it’s more about you grossly overestimating the capabilities of “human” software developers.
Web browsers are one of the most complex pieces of software. Firefox being one of the main 3 out there, and with its own rendering engine to boot.
If you think just a dozen of top software engineers are needed to maintain a project of that level and scope… then please excuse me, and don’t take it the wrong way, but you probably have not worked in anything remotely comparable. So maybe your opinion may not be that realistic or well informed.
Plus good luck attracting the quality of sw engineer you require for a project that is purely in maintenance mode. It’s career suicide.
javiercero1,
Or, conversely you may be underestimating them. Two heads of the same coin.
You speak as though no human developers are qualified to work on this, but that’s clearly not true. There are obviously developers who have the necessary skill, the bigger problem is one of time. However like I said earlier, we’re not talking about writing it from scratch, we’re talking about maintenance and focusing only on the most important changes. It’s not rocket science. You take a feature, hammer it out for a few weeks and then move on to the next thing. A competent team of 12 can do quite a bit more than you realize.
Again with the ad hominems. Well then likewise to you 🙂
Seriously though, I’ve worked on the linux kernel, which is just as big. But the thing you keep missing is that just because it’s big doesn’t automatically make it difficult. I’ve worked on a CRM application written in VB that was far more complex than working on the far larger linux kernel or writing windows drivers. You can’t just assume that something being bigger makes it harder to maintain or modify. Good coding patterns help keep millions of lines of code manageable, bad ones can leave you struggling to cope with just 100 thousand lines. That’s the thing, to an experienced developer the size is much less important than good organization and abstractions.
There’s tons of legacy code bases out there running everything from the banks and IRS to nascar and military, etc. It’s not necessarily fun work, but it’s fairly reliable work if you are willing to do it.
We keep going in circles because I don’t think you’re realizing the ridiculousness of your claim.
I am not saying that Firefox is “difficult” (whatever that means) because it is large. I am saying that Firefox is both complex AND large. The code base is 20+ million lines, it’s multi platform, the source is written in several different language, and it uses a bunch of 3rd party backends and plugins.
Anyone with moderate experience in production software of this caliber would laugh at the notion that a team of a dozen “well motivated” engineers would be able to maintain such code base. The volume of bug reports and issue trackers, per single release, that a product like Firefox generates is not trivial; even without any new features, a team of just a dozen engineers would be overwhelmed within the first couple of weeks just by the volume of tickets alone.
BTW, did you truly expect me to believe that a VB application is more “complex” than an OS kernel or a non-trivial device driver? Come on.
crystall,
Well, I do think a small talented team could do it, however I for one don’t see how they could get a profitable or sustainable business model. And on top of that even if they committed to give it away for free, it would be very unlikely that they could take market share away from power incumbents. Even microsoft, with their decent edge browser and all the resources in the world just couldn’t grow marketshare.
Regarding the lines of code, I tried to do an analysis, however it turns out there’s a lot of 3rd party code in mozilla’s source dumps that I don’t have an easy way to filter out. Some of it is in the “third_party” directory, but there’s a lot more copied throughout the firefox project. There’s plenty of google code in there, multimedia codecs, and I even found three separate copies of sqlite (which itself is a quarter million lines). I’m not trying to criticize mozilla here, it’s great that they take advantage of open source, but it’s important to note that mozilla isn’t the author & maintainer of all of it (much like I’m not the author of all 24M lines of code that I use in my linux distro or the code from hundreds of other projects that I’ve built & modified from open source). This is the way a lot of FOSS development works and even proprietary projects too. One of my clients has proprietary code from the 1990s. No one who wrote the original code is around and it got dumped in our laps. Million lines seems daunting (and it can be), but in practice you’re only changing smaller chunks of it as needed, not the entire codebase! With this in mind, it starts to make sense how a few developers can be in charge of maintaining such big projects.
The issue is not “maintaining” as it’s normally understood in the software development world (aka ensuring things continue to run, security holes are patched, bugs are fixed, etc), but rather maintaining feature parity with the other big three browsers. The web platform continues to evolve at an alarming rate, for better or worse, and this means that any browser that is only “maintained” can effectively be declared dead, because web devs won’t touch it with a ten foot pole, and sooner or later it won’t be able to display web pages as they were meant to be displayed. You can verify this statement by downloading a ten-year-old version of any of the major browsers, and trying for instance to participate in a browser-based Skype or Teams video chat – it won’t work because WebRTC isn’t implemented. And this is just one small example, the tip of the iceberg. Every new feature needs to be tested thoroughly for regressions, etc, which again if you want to compete with the other top browsers for market share needs to be flawless. Performance needs to be continually improved, again because Google et al are continually improving. On top of all that, user-facing features may not seem like a bit deal comparatively, but just in terms of the built-in developer tools, maintaining parity with Chrome is a hell of a task, where Firefox is still lagging to be honest – and if devs don’t use it, they won’t recommend it to their less tech-savvy friends, and the trend will continue tipping in the other direction.
*That’s* why it takes a hell of a lot more than twelve developers to keep Firefox competitive..
One small team can maintain the Firefox “code base”, mailing lists, and bug management (i.e.: devops). The rest would not even be remotely possible with 12 people.
It is really easy to miscalculate the effort that goes into well running systems. The browser will require specialized teams on many different areas.
Just some that randomly comes to mind: HTML rendering, bookmark management, UI toolkits, HTML canvas, JavaScript, multimedia, real time communications, network access, caching, privacy and security, and so on.
And this does not include auxiliary systems like, portability, package management and Windows installers, graphics design, marketing, addon repository management, and many many others.
Each one of these is a 4 to 12 people team, sometimes more (with sub divisions). Yes they can slim down to a skeleton crew and *hope* community contributes their expertise on individual topics. But realistically that would never be enough.
sukru,
I understand that, but I’m sure I’m not alone in having a highly multidisciplinary skill set. I’ve worked on all kinds of things in my career and I honestly think I’d be able to work on any and all of those things if the need arose. It would certainly require time to get acclimated to these subsystems, but none of those are insurmountable especially with a dozen talented developers like I said. I guess a lot of people here have full time jobs with specific roles aren’t used to small teams where they have to where many hats out of necessity. You adapt. I think it’s easy to become complacent in a large organization, but as they say, necessity is the mother of innovation.
Moochman,
Unfortunately that’s true even today. A lot of web devs only test under chrome, and for better or worse some websites fail today under firefox and I’m forced to load them up in chrome/chromium/etc. I hate that, but it’s always been this way, the main difference is who’s on top and by how much (hint sometimes the inferior browser had more marketshare for reasons that were unrelated to browser development). Marketshare is a big challenge, and I agree if mozilla looses critical mass it is in big trouble.
Ironically I’ve had the opposite issue, where some of the original webrtc projects that used to work have stopped working. Also I made use of HTML5 audio in one project and that stopped working. They break the specs by constraining the context where certain features can be used. Also with addons… damn it mozilla. Mozilla seems to have gone through a phase where they weren’t content until they broke things, I’ve been annoyed with some of mozilla’s changes over the years, but I digress…
I don’t disagree with that. It think mozilla is struggling to compete today, less money is just going to make it harder. Like Thom, I don’t know how to solve their problems either. Who knows though, if it eventually were to collapses like netscape did back in the day, maybe a smaller more efficient team could rise out of the ashes with it. Although this time they’d be competing with chromium, which is already open source and supported, so it could be a harder sell. I want firefox to remain viable, we’ll have to see what happens,
Alfman,
I believe you can do any one of these tasks if you want. However it would not be possible to work on all these tasks at the same time, nor quickly switch between them.
There is going to be a quite long ramp up time.
Let’s say you received bug reports telling buttons are rendered 1px to the right on MacBook 2018 retina models, if there is a 120% zoom. At that time you were focused on improving bookmark sychronization.
How quickly would you be able to pick up that device, figure out how the rendering system works (I can’t even read my own code from 6 months ago without some effort), dust off some books on GPU rendering (it turns out this issue was related to hardware accelerated canvas layer below), try to replicate the issue, fix it, update unit tests, deploy the new binary, … and then switch back to your context on bookmark sync, and try to remember what you were doing?
I don’t think this would be anywhere scalable for a billion user, multi-million line of code, multi-disciple project like Firefox.
sukru,
It depends, some of us do switch gears on a very regular basis. I’m really not kidding about people in smaller teams wearing more hats, haha.
Yes, I agree. Ideally you’d have some people who are already familiar with the code. If not it would take some time to familiarize oneself with things.
Wow, this reminds me so much of a bug I had to solve on the 1990s code base I was talking about earlier. 🙂
Some people are just very good at tracking down anomalies in unknown code, sometimes without even any source code! If you’ve got a hacker mentality, you don’t scare away from finding a needle in the haystack. It is a skill and sometimes an expert debugger can track things down even faster than the people who wrote the code. When it comes to debugging, sometimes knowledge of how it’s supposed to work leads to false assumptions and hides how the code is actually working.
It’s very similar to the way you and I can write a few paragraphs and not catch mistakes even on a second read because our brain is still processing what it thinks we wrote rather than what we actually wrote. However once you forget what you wrote, suddenly the mistakes pop right out (“how did I write that when consciously I know not to make this mistake”).
Sure there’s only so much you can do, however I don’t feel that you are appreciating just how much a small team can accomplish even with a big project. Anyways, I’m not really here to change your mind, just to share thoughts and ideas, and I think it’s been a good discussion. Thank you 🙂
300 engineers @$130K/yr (including benefits and taxes) works out to $40 million/yr. These aren’t SV salaries but it’s not bottom of the barrel offshoring either.
A browser is no longer only a rendering engine. In fact it had not been in many decades (even thru early Xul / Firebird years).
(Note: I work for Google but not for Chrome).
They have the Browser, several mobile versions, the VPN, email client (which definitely needs an update), and spearhead many development tools, like Bugzilla and Rust. Additionally, they have to develop and support backends like Sync, retain or outsouce web developers, hire customer support staff and so on.
[ more at: https://en.wikipedia.org/wiki/List_of_Mozilla_products ]
The current employee count is listed as 750: https://en.wikipedia.org/wiki/Mozilla_Corporation, which is after the layoffs. In fact, that is very low for all the great work that they do. For a contrast, another “billion customer” non-profit Wikimedia foundation has 400 employees: https://en.wikipedia.org/wiki/Wikimedia_Foundation, and they mostly maintain backend systems.
>This has been tried before. KHTML. Didn’t work.
He’s not talking about developing their own browser & rendering engine. KHTML was also very successful, in a way.
It sucks that Mozilla had to shed 25% of its staff, and too some long time employees, those who work on Servo, MDN, etc. Not sure what’s going to happen for my favorite browser. I hope I wouldn’t have to switch to a ‘chromium-based’ browser full time that would suck. Or maybe use something with GNOME Web (epiphany) which runs on WebKit.
motang,
Firefox has the distinction of being mozilla’s most important front-facing application, I would imagine the layoffs are mostly going to impact lessor-known projects first, so I don’t think the current layoffs will affect firefox itself. But in the future who knows what might happen come 2023 once this contract runs out.
Well they laid off the Servo research project team, that is going to affect future evolution. More impactful was they they laid off threat assessment, leaving them with only the security bug fixing team. I fear that the result of that will show up in the browser before long
Oh and developer tools, like we needed that to fall more behind…
jockm,
I hear you man. I feel for mozilla, but I don’t have the answer. Corporations at the very top are taking in trillions and the rest of us are fighting over shrinking scraps 🙁
Short of leaders who are both willing and able to put corporations in their place and cut monopolies down to size, I really think extreme concentrations of power is the end game for capitalism.
They laid off the entire Servo team??? Is there anyone left who knows the code?
As a matter of fact, if you read the internal memo, it sounds like most of the cuts relate to Firefox development, while additional investment is being made in the area of new products in the hope that they’ll turn a profit someday.
https://blog.mozilla.org/wp-content/uploads/2020/08/Message-to-Employees-Change-in-Difficult-Times.pdf
This is the kind of maneuvering you see time and again when companies are in crisis mode, and while I understand the thinking and respect the limited resources, I can’t help feeling like reducing investment in the core product is not going to help.
Moochman,
Thanks for posting! I didn’t get the impression they were mostly cutting firefox development, the “focusing firefox on users” suggests they were refocusing on user priorities that they hope will result in browser growth. It’s easy to see there are desperate for revenue though and they need some products to be profitable. I really wish em luck.
Well, the heading “focusing firefox on users” seems to me like a bit of a euphemism: the contents of that section only mention reducing investments in various areas, not increasing investment in user experience or anything like that. If you count web developers as users (which I do since I am one) it’s actually reducing the focus on those users, because it’s reducing investment in the platform features and the dev tools.
I don’t know about anyone else, but the first things I do when I install a new OS is change the search engine in Firefox to DuckDuckGo and install ublock.
But cool if Google wants to spend the money to support my favorite browser, even if it coats me a couple clicks on fresh installs.
As of note, Chrome / Edge, etc actually are slacking off on SSL requirements that Firefox is not. Chrome only warns silently about it (you have to dig to see where the SSL bit depth is). Firefox is still right by the URL.
Just as I predicted… Google wants Firefox to exist for at least two reasons: To have some token competition (that is financially dependent on them and hence doesn’t complain about the premium placement that Chrome receives on GMS-enabled Android devices or about the Chrome promotion in Google Search) and to maintain the complete hegemony of Google Search.
Back when desktop computing was still a big stage, the joke was that Apple was Microsoft’s Advanced R&D department. When there’s no competition, innovation stagnates and the VP or Director of R&D answers questions from the troops about big new features by saying, “I haven’t heard any customer asking for that.”
In all fairness, Apple put themselves into a corner market share-wise by selling way too high.
meh,
Even my most highly regarded OS projects are owned by Large soulless, abusive corporations. That’s the proce of “Freedom” I guess.