Homebrew: A Packagers Deep Dive

09 November 2021

A deep-dive on the interesting (both good and bad) aspects of the Homebrew package manager that will be interesting to other package manager maintainers or enthusiasts.

Based on my experience as a user (and very sporadic packager) of other OS system and language package managers I detail the things I feel that Homebrew does well, badly and what we plan on changing and what we cannot.

Presented at PackagingCon 2021.

Show transcript

0:00 Well, yeah, thanks, everyone. Thanks for having me today. I appreciate being asked to kind of nerd out on package management because it's something that I've cared about for a fair wee while. So there's three talks, I think, from homebrew maintainers happening in this conference. And I'm going to kind of do like a deep dive into homebrew and some of the stuff I kind of have seen with homebrew that's kind of worked well, not working well, a little bit of history and stuff like that. So let's have a wee walkthrough for you.
0:30 So my name is Mike McQuaid. Nice to meet you. I know some of the folks who are watching this, I'm sure I've met a bunch of you already, and I hope to meet many more. And you can get in contact with me if you have any kind of thoughts or questions about either this talk or homebrew in general, then get in touch on Twitter or through my email, which is on my GitHub or whatever.
0:51 So I've been working on homebrew for about 12 years now. And I've been the project leader, I guess, for three or so. And yeah, it's a really pretty interesting project to work on. And it's a pretty interesting package manager. So let's have a wee walkthrough these things.
1:06 So as I mentioned before, what I'm going to talk through today is the history of homebrew and how that sort of has informed where homebrew is today and some of the kind of original design decisions and stuff like that, things that homebrew does well, from my perspective, at least compared to other package managers, things that it does badly.
1:24 And then moving on from these, the things that homebrew can't really change, that it does not very well, and some stuff that we have planned for future, which will hopefully be some good changes.
1:35 So let's start off with the history. So I'm going to actually walk through, interestingly, with homebrew, it's one of these ones where the whole development from day one, pretty much, is in the open, and you can kind of review all that stuff.
1:49 So you can actually go all the way back in the Git logs. We've moved repositories over the years on GitHub, but we have been diligent in kind of maintaining a lot of Git history.
2:00 So in the homebrew repository, you can see the original commit for homebrew that was made by Max Howell, the original creator, on 21st of May, 2009.
2:10 So like this kind of readme sort of describes how the project works at this stage.
2:17 I mean, it's fairly different, even within a few months of this.
2:20 So I think I was involved, I don't know, like three, four months after he sort of did this original commit.
2:26 I sort of knew Max, he was a friend of a friend, and we both, I think we were working in London at the time.
2:31 But it's interesting to see how many of these kind of concepts are still integral to homebrew 12 years on.
2:39 So I'm not going to read through everything here.
2:41 And if you can't read all the text here, then A, you can go and check out this link and see the text there if you want to.
2:48 But B, also like, I'm just going to gloss over stuff and you can go back and get back to it later if you're interested.
2:54 So the first thing here that's sort of notable is this kind of idea that, like, say, if you're still in WGET, WGET's embedded within a version directory.
3:02 So you've got WGET slash 1.14 bin WGET.
3:06 I think that was inspired originally by Gobu Linux, which does something similar.
3:10 It might have been something that did that before Gobu Linux.
3:12 But this idea that every software and every version of the software is in an individually versioned prefix, which are then symlinked back so that you can kind of, you know, use a, you know, user local bin WGET, but that actually points to user local seller WGET 1.14 bin WGET, for example.
3:29 So this is still remains the same and as Max says, you kind of use the file system of the package database, which is kind of nice because it's less preserved state and we don't need to implement like a database package store or whatever.
3:40 So the next thing, again, the stuff that remains the same today that's sort of interesting with the original approaches with Homebrew are we kind of still try and use sort of dependencies from a macOS system when they're available for us.
3:51 We also still don't run as root.
3:54 And there's a few exceptions where a little bit more liberal on than we used to with stuff like running in Docker containers on Linux.
3:59 And we can still be installed anywhere.
4:01 But I guess as I'll talk about in this talk, basically almost anything you could do with Homebrew back then, you still can do with Homebrew now.
4:10 But the difference being there's paths that we support now compared to before where it was, you know, arguably everything was supported and nothing was supported.
4:19 So moving forward, like Homebrew doesn't really work quite the same way as this, you know, the really first versions were literally you were just calling a Ruby script and you're executing with Ruby, whereas now we have this kind of brood command.
4:30 But again, still you can see most of these operations kind of seeing the installed size of a package on disk by using typical Unix file system utilities and stuff like that.
4:40 So there's even a little touch on here where Mac sort of debates whether or not we should add a brute command, but says that, you know, if you don't add it, then you'll understand that the system is better.
4:49 And obviously we did have a brute command eventually.
4:53 And I think there's still a certain amount of understanding, but I think it shows kind of the real technical focus for the original Homebrew users were expected to kind of, you know, know the way around the Unix command line very well and not just use it as a tool for installing stuff, which might be the first Unix command line tool that they use.
5:10 I still think there's that middle paragraph where it talks about with AppGate, you don't really know what's happening with Homebrew you do.
5:16 Obviously, despite the fact we have like a brew and still come on now, our kind of formula, which is what we call our package descriptions in Homebrew are still fairly readable.
5:24 If you look through, it's fairly, there's not, there's a little bit more magic than there used to be, but it's still fairly obvious like what commands have been run under the hood to actually make it work.
5:33 So there's some kind of opinions here about like Mac ports, which at the time was the kind of main competitor, I guess, that Homebrew was kind of seeking to replace.
5:42 And, you know, we've both found their own little individual niches that kind of make sense for Mac ports and Homebrew.
5:48 And this sort of, you know, Mac ports is installing its own LibZ, OpenSSL, et cetera, stuff like that.
5:53 Like Homebrew still tries to take this methodology where we can.
5:57 Unfortunately, with stuff like OpenSSL, we've got to the point where compared to back in the Leopard days, the system OpenSSL was so outdated at this point.
6:05 Well, it's Libra SSL firstly, but it's, you know, it doesn't work for a lot of things that we need it to.
6:09 So unfortunately, we have to rely there.
6:12 But we still, you know, we still try and use what we can from the Mac OS system when we can.
6:16 And also the stuff that Homebrew will never build.
6:18 So, I mean, all of these have been broken in some form.
6:20 So, Katie, you know, yeah, we don't package that stuff in Homebrew proper, but something else does.
6:24 There's various taps that kind of you can use, taps being the third party Homebrew repositories that use these type of things.
6:32 Anything that can be distributed in an app bundle, again, sort of.
6:35 So Homebrew can install those, but that's kind of separate to formula.
6:39 That's Homebrew Cask, which was then merged into Homebrew itself.
6:42 So you can brew install Google Chrome, but that's kind of using a completely different way of doing that.
6:46 Anything that you just saw outside the Homebrew tree, same deal.
6:50 Stuff in the Homebrew tree is done by formula.
6:53 Stuff that needs to go outside the Homebrew tree is done by casks.
6:55 And stuff, OS X, now Mac OS, already does.
6:58 RubyGems, duplication sucks.
7:00 So, like, we do a little bit more duplication.
7:02 We will now package some stuff in RubyGems, NPM, whatever, but generally only applications
7:07 if you don't care about the fact that they're written in JavaScript to Ruby or whatever it may be.
7:12 But we don't go and take the, say, Debian approach, or at least Debian approach, over a few years ago
7:17 and go and try and package, you know, every library that we use for an NPM application or whatever it may be.
7:22 So, compiling from source is an interesting thing.
7:26 So Homebrew is essentially still under the hood, a from source package manager,
7:29 but we built binaries, which we call bottles, on top instead.
7:34 So now we actually only support building binary packages and users consuming those binary packages.
7:40 We have a build from source fallback, which is what we are using in CI to build those binary packages,
7:45 but that we don't really support that.
7:46 It's, you know, over the years, we just found it was far too error prone to support, like, from source package installation.
7:54 And, again, like with Homebrew being originally a macOS package manager,
7:57 we wanted to kind of keep the focus on stuff that works out of the box.
8:02 We look kind of like Apple does.
8:03 We would rather offer less knobs to twiddle, but a system which works better most of the time.
8:10 So, I guess, of course, that's very debatable whether Apple works better, but it's my talk,
8:15 so I guess I can say what I want.
8:16 How do we advertise to someone who's packaged out as eight?
8:19 Again, this points to the fact that Homebrew from the outset was using GitHub very heavily.
8:22 And this is, again, something we still do to this day,
8:25 and the kind of contribution focus in Homebrew is still very, very strong.
8:29 And then, finally, the thing people always ask,
8:31 Homebrew obviously has this beer theme that started from the outset.
8:34 And, yes, Homebrew was originally advised under the influence of alcohol.
8:39 So, these themes, which I think still apply with Homebrew today,
8:42 about avoiding duplication, a fairly focused scope,
8:45 not doing things necessarily other package managers might for you,
8:48 community ownership of the package manager rather than it being entirely done by Macs
8:52 or the maintainers, and making contribution very easy.
8:54 I think those stand on and set the kind of foundation for where the package manager is today.
8:59 So, this segues nicely into the things I think Homebrew does well.
9:03 So, going back to that, contributions.
9:06 So, we've had over, I guess, between the two repositories,
9:09 probably over 10,000 contributors to Homebrew at this point over a 10-year period.
9:13 People have asked in the past, like, how have we done that?
9:17 How have we got so many contributors?
9:18 And I think part of that is making it very easy.
9:22 So, we have a community that's very welcoming to people.
9:26 There's sometimes an expression that I've heard people say where they talk about, you know,
9:32 the drive-by contributors and pull requests that people just submit one,
9:37 and then they forget, and they never come back to the project.
9:38 In my mind, like, if that is not useful to your project,
9:42 then that's a sign that you set expectations maybe a little bit badly
9:44 of what your project wants and what they don't.
9:47 We have a lot of people who have only contributed once,
9:48 but it's been really useful.
9:50 And we have created dedicated tooling to make it much easier to do this.
9:53 So, in our documentation, how to open a pull request,
9:56 the most common thing people were doing, we would find,
9:58 would be submitting a pull request to change a version.
10:02 We call it bumping a version.
10:03 So, increasing the version of a package to the newest version that's come out.
10:07 So, we have a dedicated tool, BrewBumpFormulaPR,
10:10 which you can specify a package name and the new version and stuff like that,
10:14 and it will actually not only make the changes for you locally,
10:16 commit them to Git, but it opens the pull request for you.
10:19 And it's pretty cool to be able to see people being able to create pull requests
10:22 without really any understanding of Git,
10:23 without really any understanding of what Hover Brew is doing underneath the hood,
10:26 but just following steps and documentation,
10:28 and that's their way of getting involved with the community.
10:30 And it's actually useful for us as well,
10:32 because that saves us kind of from having to check this stuff manually.
10:36 Stepping onwards from different stuff that Homebrew does.
10:38 So, another thing I think we do fairly well is doing things for the user.
10:42 So, in this screenshot, you can see I'm upgrading a package here,
10:47 so upgrading shell check,
10:48 but this is actually doing a couple of things,
10:50 which had to be done somewhat manually.
10:52 So, it's updating Homebrew itself,
10:55 effectively running a brew update command for you before it does anything.
10:58 So, this was a change that we made a few years ago
11:01 to basically reduce the amount of support requests we were getting,
11:03 because we were getting tons of support requests,
11:05 which were 90%, probably 95%, maybe 99% were resolved by run brew update and try again.
11:11 So, this basically means that we will talk about by default, people are always getting the latest version of Homebrew and the latest version of the packages.
11:19 And this makes more sense from our perspective, but I'll come more onto that later.
11:24 So, this is updating Homebrew.
11:25 It's upgrading the package for you, downloading it, etc.
11:29 and then it's running brew cleanup afterwards to kind of clean up the old versions of the package and the kind of old cache files and stuff like that.
11:37 Again, we didn't use this by default automatically for people, but Homebrew ended up taking up more and more space and your average user didn't understand why.
11:44 So, another thing Homebrew I think does really well, I'm biased, of course, because I am a homebrew maintainer,
11:49 but we really prioritize the kind of health and mental health of our maintainers.
11:54 We try and we've got documents on how to avoid burnout and we try and set the expectation in general that in the project,
12:00 that if you have to choose between a maintainer suffering and a contributor suffering or a maintainer suffering and a user suffering,
12:06 and those levels of suffering are roughly equal, then we are always going to prioritize the maintainer.
12:12 Because ultimately, at the end of the day, the project without any maintainers ceases to exist, all activity stops,
12:17 and we don't have an open source project anymore.
12:19 And assuming that every maintainer, as the maintainer guide to avoiding burnout says, is using Homebrew,
12:26 then we can't have no users, because so long as we have one maintainer, we have one user.
12:30 Whereas, if we only have users who aren't contributing and who aren't maintaining the package manager,
12:35 then we don't have a package manager anymore, effectively.
12:38 So, yeah, so this is something I would advise you to do.
12:41 I personally have taken some quite strong leads on this in the past.
12:44 You can check out this blog post that I recommend if you're a package manager maintainer,
12:49 I recommend you read this and really internalize this,
12:51 because it's really hard to get your head around the idea that you don't actually owe anyone anything.
12:56 When it comes to open source, everything you do is from the principle of charity,
12:59 and that's great, and you can do that, and it's very nice to do things,
13:04 and there are things which I feel obligated to do, and I do do them,
13:07 because I want to try and be nice, but you don't have to do them.
13:10 And particularly when people are being entitled jerks, you do not have to do them,
13:13 and you do not have to fix legitimate bugs from entitled jerks.
13:17 So that's me on my soapbox briefly.
13:19 But I do think this focus on, for a relatively long period, on kind of maintainers,
13:24 some people don't like it, but the maintainers in the project seem to like it,
13:28 and it seems to keep people around for a reasonable amount of time
13:30 and allow them to have a good time on homebrew,
13:32 which means they spend more time on homebrew.
13:34 So I guess my summary of stuff we do well, community ownership of the project,
13:40 it's not just owned by the maintainers.
13:42 We expect our users to get involved, but we make it easy to contribute.
13:46 We have fairly sensible defaults, so we don't expect you to kind of just know
13:49 to run a command after you run some other command.
13:51 We either tell you to do it, or we do it for you,
13:54 and then we prioritize maintainers.
13:56 Things we do badly.
13:57 So, wait a second, this is the same.
14:00 Oh, no, aha, this is me being very clever here,
14:03 because like in most software development, there are trade-offs.
14:06 So these are the things that we prioritize in homebrew,
14:10 but you could equally, and some people do if you read comments
14:13 and places that hack and use, flip this around to the other way around.
14:15 So instead of community ownership, you could say,
14:17 well, our maintainers don't take sufficient ownership.
14:20 Instead of user contribution being a good thing,
14:23 we could say, well, we punt things off to our users
14:25 rather than doing them ourselves.
14:26 Instead of good defaults, we could say, well, for power users,
14:29 the defaults are bad, and also the defaults keep changing over the years.
14:32 And instead of saying we're prioritizing maintainers,
14:35 they're saying, well, homebrew is just trying to do all they can
14:37 to kind of close issues and use stale bots and lock bots and whatever.
14:40 So I think these are valid criticisms of the project.
14:44 These are valid trade-offs that I think we're making,
14:47 but it's worth pointing out what these trade-offs look like.
14:50 So this is an example of a fairly normal issue that's created
14:53 to where someone said, okay, I instilled this package using homebrew.
14:56 But in homebrew, in our case, we don't have,
15:00 we don't maintain issues, and we don't try and fix packages
15:03 that we have which are like broken,
15:06 which are basically outweighed of our responsibility.
15:08 So we've got two maintainers here,
15:09 Bo98 and S MillerDev, who are saying, yeah, upstream to fix this,
15:13 and we're going to close this
15:15 because homebrew is not going to keep track of this indefinitely.
15:17 Similarly, on a pull request where someone says,
15:21 oh, this thing fails,
15:22 Sean Miller is saying here, yeah,
15:26 could you make a pull request to update this?
15:28 Because the person there seems to be the one affected,
15:30 and they seem to be the one who knows what needs to be changed.
15:32 So it makes sense to Sean and to me
15:34 that that person would submit a pull request to make those fixes.
15:37 So on to default.
15:39 So everything on this slide,
15:41 so analytics, auto-update,
15:43 boot snap,
15:44 checking dependence,
15:46 cleaning up formula after installation,
15:48 and this slide,
15:49 not using the GitHub API,
15:51 no insecure redirects,
15:53 not doing cleanup after install,
15:56 and not automatically upgrading install outdated formula.
16:00 Like various people over the years have argued that,
16:03 well, sorry,
16:05 in homebrew,
16:06 none of these used to be the case.
16:08 I think with a couple of exceptions,
16:10 these used to almost all be the default behavior.
16:13 And everyone has argued,
16:15 sorry,
16:16 someone has argued for every single one of these over the years
16:19 that they should be the default.
16:20 But we decide the defaults that we do
16:23 based on prioritizing the 99% case,
16:26 the people who use homebrew most often.
16:28 we want homebrew to install things,
16:31 fix things up.
16:33 If we think that we may have broken something,
16:35 we would rather do an upgrade that was unnecessary,
16:37 or reinstall that was like detected,
16:40 if broken linkage was detected or whatever,
16:43 and make an installation take a little bit longer
16:45 than you might have expected,
16:46 but produce a result that's correct at the end.
16:49 And some people don't like that behavior,
16:50 but we think that's the best for users overall.
16:53 Similarly,
16:54 we have a homebrew developer command
16:55 that you can get auto opted into
16:57 by running certain commands,
16:58 which puts you on a beta channel of homebrew.
17:00 this is very useful for our user testing,
17:02 but some people don't like this.
17:04 But like if we had an opt-in testing there,
17:06 we would have much fewer people in it.
17:08 So we would rather prioritize what's easier for us.
17:10 Similarly,
17:13 on our issues,
17:14 we now have these kind of required issue templates
17:16 with required fields you have to fill in.
17:18 No doubt,
17:19 this is a little bit harder for people to fill in.
17:21 Maybe it takes a little bit more time
17:22 and requires more information from them.
17:23 But this allows maintainers time to be prioritized
17:27 so that we can jump in
17:28 and have the information we need
17:30 when someone submits an issue
17:31 rather than having back and forth.
17:32 So this leads us on to things about homebrew
17:35 which are a bit broken that we can't change.
17:37 So one is our version support.
17:40 Our version support isn't really great.
17:41 You can go and read our documentation,
17:43 but effectively,
17:44 you're just creating multiple formula
17:45 because homebrew doesn't have the concept internally
17:48 of depending on a particular version.
17:49 So we just have a version being outdated
17:52 or it's not outdated.
17:53 And if it's outdated,
17:54 And if something depends on something that's outdated,
17:56 it gets upgraded as part of the installation.
17:58 So similarly,
18:01 installing old versions of formula,
18:02 like people are like,
18:03 oh, well,
18:04 can't you just install the old version
18:05 from the Git history?
18:06 No, sadly not,
18:07 because that stuff breaks all the time
18:09 because we have to make changes
18:11 as macOS changes and the like.
18:14 Our installation process,
18:15 people would love it
18:16 if we supported
18:17 the non-default installation parts.
18:19 But if you use the non-default ones,
18:21 you don't get binary packages.
18:22 And if you don't get binary packages,
18:23 we're not testing what you're doing in CI
18:25 and things may be broken.
18:27 So we don't support those
18:28 because we don't have
18:29 the volunteer time
18:30 to be able to do so.
18:31 I'm almost at the end,
18:34 so I'm just going to fire through quickly.
18:35 Our terminology,
18:36 similarly,
18:37 we can't really change this at this point
18:39 without confusing existing users.
18:41 We're written in Ruby.
18:42 There's upsides and downsides of that,
18:43 but it slows things down a little bit.
18:45 So we can't really rewrite Homebrew
18:47 not in Ruby.
18:48 And particularly our DSL
18:49 is very heavily reliant on Ruby.
18:51 But stuff we are planning on changing
18:53 in the nearest future.
18:54 So we plan on allowing formula
18:57 to be installed
18:58 without having to have Homebrew core
19:00 or Homebrew cask
19:01 checked out on disk.
19:02 So they take up a lot of space
19:03 and they take a long time
19:04 to fetch and clone
19:06 and update.
19:07 So that will be a big win
19:09 when we're able to ship this by default.
19:10 But you can try this out now
19:11 with this Homebrew install from API flag.
19:13 We hope to have more
19:15 binary packages be relocatable,
19:16 which means you can install
19:18 and they will be used.
19:19 But if we ever get to a point
19:21 where 100% of our binary packages
19:23 are like this,
19:24 to install Homebrew anywhere,
19:25 which would be great.
19:27 But you can check out
19:28 what stuff we have planned
19:29 from our help wanted labels
19:31 on our repository.
19:32 So things that Homebrew does
19:34 like that's coming in future.
19:36 You can expect some faster updates,
19:38 hopefully some more relocated bottles
19:39 and relocatable bottles
19:41 and maybe other stuff.
19:43 Who knows what the future will hold?
19:46 So yeah, sorry,
19:47 I've run a little bit late with this.
19:48 I don't know if we have any time
19:49 for questions or not.
19:50 But thank you everyone
19:52 for coming and listening
19:53 and thank you for everyone
19:54 who has contributed to
19:55 or maintained Homebrew
19:56 over the years.
19:57 We couldn't have done it without you.
19:59 So I'm seeing some questions
20:13 I'm going to read off the screen
20:14 and answer the questions
20:15 Given that you have to package up
20:17 do you find it valuable to say
20:18 try not to package
20:19 When you move to a model
20:20 where you did that
20:21 or would it be worse?
20:22 So in my opinion,
20:24 I think it's still valuable
20:25 to use things from the URL.
20:26 the OS where they're provided.
20:27 So like Zedlib, OpenSSL,
20:28 whatever,
20:29 because it just saves time
20:30 and it means that you can defer
20:32 more of those security
20:33 things to the OS provider.
20:36 OpenSSL is an example
20:37 where we don't do that anymore.
20:38 But there's still cases
20:41 and I still think
20:42 rather than being
20:43 consistent with everything
20:44 and packaging everything ourselves.
20:51 I guess it doesn't let me say things
20:53 if I'm not on screen.
20:55 Here's the second question for you.
20:58 Oh, yeah.
21:00 And that question.
21:01 So if all the versions
21:02 are removed automatically,
21:03 to prefix the installation
21:04 by version?
21:05 Maybe not anymore,
21:07 but I think it still does
21:09 because of the existing
21:11 like foundation
21:12 of the old way things worked,
21:13 you can turn off that behavior
21:15 and then you can get
21:16 like keep versions around
21:18 and have that different behavior.
21:19 But also I still think
21:21 to be able to segment things
21:23 because that means
21:24 that if you're upgrading a package,
21:25 you know reliably
21:26 that you can destroy
21:28 everything within
21:29 the old version's prefix.
21:30 Whereas if you prefix
21:32 the installation by name
21:33 then you would have to do
21:35 some sort of swapping
21:36 back and forth.
21:37 Whereas this way,
21:38 the swapping is only done
21:39 to the version,
21:40 which can be cleaned up afterwards.
21:45 All right.
21:49 That is all.
21:50 Thanks for the talk, Mike.
21:52 And well,
21:53 if folks have more questions,
21:55 you can ask them
21:56 on the element room
21:58 that we have
21:59 or on gather.town.
22:00 Yeah, I'm going to head over
22:02 to the element room now
22:03 any other questions people have.
22:04 But thanks so much everyone
22:06 and thanks Padian
22:07 for organizing.
22:08 Thanks.
22:09 See you over there.

Mike McQuaid

Homebrew: A Packagers Deep Dive