Building Homebrew in Ruby

19 April 2019

Learn about things Homebrew loves, hates and struggles with because it is built in Ruby.

Presented at RubyKaigi 2019.

Show transcript

0:00 Hello, everyone. Konnichiwa. I'm gonna talk today about Homebrew and how we've built Homebrew
0:08 in Ruby. I'm gonna ask for quite a lot of putting your hands in the air because it helps
0:14 me be sure that you're listening and gets feedback, and also it stops you all from falling
0:20 asleep. So, anyone here use Homebrew? Awesome. Anyone here submitted a PR to Homebrew ever?
0:29 Was it merged? That is sad, and probably my fault. Does anyone like our new logo? Can
0:40 you tell the difference? It's really different from our old logo. In fact, who thinks the
0:47 new logo is on the left? And who thinks the new logo is the one on the right? There was
0:52 a designer guy who works on Homebrew who explained why this is not real, like a real beer glass
0:59 glass, whereas this is like real, and that's good. So, who likes putting their hand in the
1:04 air? Woo! Woo! Right, so I'm Mike. I've
1:10 met some of you already. If I haven't met you already, feel free to come and talk to me anytime.
1:14 I'm here till Saturday night. I've been working on Homebrew for 10 years now this year, which
1:19 is quite exciting. And I'm the project leader apparently. I've been at GitHub for about five,
1:26 well, I guess just over five years now, and I'm a senior engineer working on improving GitHub
1:31 for open source on a team working on kind of fun secret stuff that will ship in the next
1:36 few months. If you're interested in talking about GitHub and open source or Homebrew, then
1:42 give me a shout. I've got my email address there. Feel free. If you email me, I reply unless
1:47 you're being mean, and then my coworker convince me it's generally better to not reply and get
1:52 in arguments with people on the internet. So, yeah. Feedback always welcome. Going to talk
1:59 today, as I mentioned, about Homebrew and Ruby. You may or may not know Homebrew, the first
2:04 version, like the first file and commit, was written in Ruby right from the outset. I think a big part of the reasoning for that was Ruby was one of the
2:12 languages that shipped with Mac OS, and Max Howell, the guy who created Homebrew originally, wanted to build something that didn't require any dependencies to run on Mac OS.
2:21 And at the time, he had played around with a bunch of different languages, and he, much like myself, fell in love with Ruby and thought that it was a good fit for Homebrew.
2:30 But because I am Scottish, we can't start with nice positive things, so we need to start with the negatives. So, the first problem Homebrew has is a result of that.
2:42 which is the fact that, because we want to use the Ruby that's provided with Mac OS, we don't really get to control our own version.
2:49 So, who here is running Ruby 2.6 in production at the moment? You lucky, lucky people. How lucky you are.
2:57 Yeah, so, Ruby 2.6 is cool. There's lots of good stuff in there. Unfortunately, until 2016, Homebrew had to still work with Ruby 1.8. And until, like, earlier this year, our installer script, also written in Ruby, had to
3:12 that support Ruby 1.8. Is anyone, anyone here still running Ruby 1.8 in production?
3:17 Yeah, a couple of people. Shout out the people at the back who are weeping as they put their hands in the air.
3:25 Ruby 1.8's great, to be clear, but, you know, it's 2019 now. We're all meant to be in flying cars and Ruby 2. whatever versions.
3:33 So, we're running 2.3, which is the version that ships from Mac OS. Like, this is nice. It's certainly a step up from 2.0, which was the previous version we were running, which was a step up from 1.8, which is the version before that.
3:45 And if you run user bin Ruby on Mac OS, that's the version you get. And we use this system version where possible. If you have a nice new enough version, which is the current version of Mac OS or the previous one, and where you don't, this is the thing that allowed us to get off supporting old versions of Ruby.
4:03 We have our own little thing called portable Ruby. So, this is basically just our own little binary build of Ruby that's built such that we don't hard code any paths anywhere. And you can install that anywhere on your Mac OS system and it will behave nicely and just work nicely.
4:18 So, again, we still like to use the system stuff where possible. So, although in theory, we could just use this to ship our own version of Ruby 2.6 for everyone, it's a little bit nicer to rely on the system where we can.
4:30 Unfortunately, there's a few bits of weirdness with this. A, like people, i.e. me, have to remember to compile and build this for when we want to kind of get the versions compatible, and that's a bit of a pain and a bit too manual a process for my liking.
4:44 And also, there's weird stuff I don't understand that I could probably understand this week if I talk to enough people about if you have gems installed with the portable Ruby version and then you install the, sorry,
4:57 sorry, you remove that and you have an upgraded Mac OS version or whatever, then they don't play nicely and the native extensions will start sobbing blood and all that type of thing.
5:06 So, and something else I discovered today, which made me sad. So, I've been inspired to do a couple of things through Ruby with Kaiji.
5:12 And one of the things was like, right, I'm going to finally get this frozen string stuff sorted.
5:15 Uh, and I was like, kept banging my head against the wall with some like ERB issues and noticed this nice bug report, which has now been fixed by cropped that bit out of it.
5:25 And it's been fixed and was originally reported on 2.4 and didn't have a bad port to 2.3, unfortunately.
5:32 Uh, so I could request one, but who knows whether Apple will actually ever include that or not.
5:37 So, at the moment, I'm going to just live with having 732 comments at the top of files saying frozen string literal equals true.
5:46 Oh well.
5:47 Uh, so one of the nice things about kind of living in the Ruby ecosystem is although we're a little bit different and we're own special snowflake of an application,
5:57 Um, we do rely on a lot of the kind of tools that are commonplace.
6:01 So, if you use IRB like us, put your hand in the air.
6:05 If you use, anyone use Ron?
6:08 Or, yeah.
6:09 Uh, so Ron is a nice little gem for generating man pages amongst other things.
6:15 Um, so we use that for generating our man page from a markdown file.
6:19 Uh, and ERB, which is one of the things that was broken.
6:23 Oh no.
6:24 Uh, Ruby prof, anyone?
6:26 Nice little profiler.
6:28 Uh, you may well use something better that I'm not aware of.
6:32 Um, so we basically have our own little wrappers for all these things because, uh,
6:37 because Homebrew runs its own Ruby processes and everything has to be initialized with a bunch of environment variables,
6:44 which get kind of set from bash first of all.
6:47 So then you kind of need these like little hooks that are basically just little shims around installing gems and things like that.
6:54 And we'll come onto it later.
6:55 Um, and then executing the Ruby, the Ruby processes that Homebrew expects in the right way.
7:01 Um, and people use our spec.
7:04 Anyone heard of that?
7:05 Unusual little thing.
7:06 Yep.
7:07 Quite a few people.
7:08 And anyone use the bundler install standalone mode.
7:11 This is a relatively recent discovery of mine.
7:13 And it's quite cool.
7:14 We'll talk more about that now, in fact.
7:16 Uh, so we do use bundler now to install our gems.
7:20 We used to previously have this weird ghetto version of people just like downloading stuff and untarring it.
7:28 And then like just adding on GitHub and no one knew what version anything was.
7:32 And it was very, very sad.
7:34 Uh, basically based around the initial thing where we didn't want Homebrew to kind of have to rely on users to manually install gems.
7:42 And in general, we don't want Homebrew users doing not development work on Homebrew to even care that Homebrew is written in Ruby.
7:51 So a Homebrew user wanting to just install some software should not need to install any gems.
7:57 They should not need to know about Ruby gems or bundler or whatever these things are.
8:01 They should just be able to do that.
8:02 And it's only when you kind of get a little bit deeper into kind of developing Homebrew or developing a Homebrew formula,
8:07 which is what we call our kind of package description files that you need to care about any of that stuff.
8:12 Uh, I also learned about gel, which was, uh, this tool that was released yesterday.
8:17 That's kind of a much speedier version of bundler with kind of fewer features.
8:21 So I've been playing around with that to see whether we could use that to speed some stuff up as well.
8:25 So as I mentioned, because we don't want to have, uh, users having to manually learn this stuff.
8:33 Like we don't want to have people have to do this.
8:36 Uh, this would be a bit of a pain for, again, the reasons that people have to manually install bundler and do that jazz,
8:44 but also like it slows down the invocation of the Homebrew process.
8:48 And we want to try and make things nice and snappy where we can.
8:51 We have a single gem file while we actually have,
8:54 we're down to just two gem files in our, uh, repository.
8:58 Now we had four a few weeks ago.
9:00 Um, so this includes, as you might expect,
9:03 all the stuff that we need both at runtime, uh, but then various development tools as well.
9:08 And then this is the magic file that bundler standalone generates for you.
9:12 Uh, so it sticks it in here for us.
9:15 And then we have under library homebrew vendor bundle, Ruby.
9:19 That's where we have like all our gems that we commit into the repo.
9:23 Um, and then what this file gives you, which is pretty nifty is if you haven't used it before,
9:28 is basically a bunch of lines get spit out, which look like this,
9:32 which is basically adding stuff to the load path based on the version of it.
9:37 the version of the gem and the version of Ruby and all that type of jazz.
9:41 So basically what that means is that you can even turn off Ruby gem support
9:45 as I'll come along to later in your version of Ruby that you're running.
9:49 Um, and you don't need to have bundler installed on the system at all.
9:52 And then with this file,
9:54 you can basically have it go and do all the magic and include all the gems at the right versions
9:58 and stuff like that without you having to worry about that.
10:01 Um, so I found that a really nice way of effectively kind of making it easier to run projects.
10:06 And just get all your gems in place without needing to kind of have the,
10:11 the runtime overhead of using bundler for stuff.
10:14 I guess it's worth noting.
10:17 Again, I keep talking about kind of the runtime overhead.
10:20 That's because we're, I guess, in the relatively unusual case in 2019,
10:25 in that we are still primarily a Ruby project,
10:28 which does not operate on or inside a web request.
10:33 Uh, we are still like just a command line application that just happens to be written in Ruby.
10:37 So for us,
10:38 it's really important to basically spin everything up and get to doing work as quickly as possible.
10:44 Whereas if you're doing this in a rails project,
10:46 like this would probably be pretty dramatic, premature optimization,
10:51 because you're going to start your web server like once per deployment or whatever it would be.
10:56 And you don't need to kind of repeatedly do this again and again and again.
10:58 So all this kind of overhead becomes not something you probably need to care a whole lot about.
11:04 But it would be nice if we could kind of check in all this stuff to the repo,
11:09 but because we don't want to check binaries into the repo,
11:12 because of the slight incompatibilities I mentioned earlier between the Ruby versions we support,
11:16 we don't check everything in.
11:18 So like native extensions that get built, those don't go into the repo.
11:22 But we manage to, with the gems we're using,
11:26 we don't actually require any gems with native extensions for doing, again, user stuff.
11:31 It's only when you get to developer stuff.
11:33 So for example, if you're using a brew style that does our style checking,
11:38 I'll talk more about later,
11:39 then that's going to install Rubicop, which needs some native extensions.
11:43 If you're going to do like various other bits and pieces that are kind of more developer-y
11:47 and have to compile some secret underneath the hood,
11:50 then that is going to have to pull in bundler,
11:52 which then installs all this stuff for you.
11:54 And then that's just like a nice little one-time thing.
11:57 And then after that, you have everything that you need.
11:59 And due to all this stuff,
12:01 we now have the ability to use active support stuff inside Homebrew.
12:06 Who likes active support string present?
12:08 I realize it's maybe a divide.
12:10 Oh, not a lot of people.
12:12 Well, I like it.
12:14 So there.
12:15 So yeah, basically, it's now much easier for us to use particularly pure Ruby gems inside
12:21 Homebrew to kind of do a little bit more code reuse than we previously were.
12:25 When previously, it was you find some cool gem that would be useful for doing stuff in Homebrew,
12:30 and the pain of vendoring that and any dependencies is high enough that you just didn't bother.
12:35 So we had like one completely standalone gem for doing like PList reading or something.
12:41 And then like, I think one internal gem that one of the homebrew maintainers wrote to kind of read
12:49 MacO files.
12:50 And which are, it's like the Elf format equivalent on Mac OS.
12:54 And but aside from that, we basically just didn't have anything.
12:57 Whereas now we have active support.
12:59 So as I'm talking, I guess about gems and our development dependencies, a special mention goes
13:06 to RuboCop because we use it pretty widely now and in some cool ways.
13:10 So if you run our brew style command, that's going to go and check over the Homebrew codebase
13:17 and basically look for kind of style violations and stuff like that with RuboCop.
13:21 It's mainly kind of part of our CI job, but it's also like a little thing that we can go
13:25 and tell people to run a single command to basically check everything's fine before they submit a PR.
13:30 But if you specify a formula, again, that's the kind of package descriptions,
13:34 then you get Homebrew specific formula checks as well, which are nice.
13:38 So because they've all been written in RuboCop and because again,
13:41 the way that we tend to and try to do things, we don't believe in having to,
13:47 I guess the sensible proper probably way of doing this would be saying,
13:50 okay, well we make a Homebrew RuboCop gem that people install and then they get all this cool stuff.
13:56 But we want all this stuff to work as well as possible with no intervention at all.
14:01 So because the source code for these files is under a tree hierarchy that we control,
14:07 the really nice thing is if you install, say, any sort of editor like this is VS Code,
14:12 which is my current one of choice, and if you install the relevant kind of RuboCop plugin,
14:17 if you start like editing a file here, like I'm editing the wget formula,
14:21 I've moved around the dependencies here, so I put them in the wrong order.
14:26 And because basically we've tried to turn all maintainer pedantry into actual code checks,
14:33 because it's A, a pointless waste of time to have maintainers making the same pedantic comments again and again,
14:41 and B, if you encourage maintainers to instead of being pedantic on PRs to actually write some code,
14:46 like 90% of the time they'll decide that it's not actually important enough to warrant writing the code in the test for that,
14:52 and then you kill two words with one stone.
14:54 So here I've moved around the build dependency, it wants that to be above this dependency instead,
15:01 because that's just the ordering we have here.
15:03 So it's printed out a nice little error in here.
15:06 So the cool thing about that, as I said, is that doesn't actually require any intervention.
15:10 If you were just contributing to HomeRoof for the first time,
15:12 and you are a Ruby developer and had this set up in your editor of choice,
15:16 then what's gonna happen is this is just gonna run without you having to do anything,
15:20 and it's gonna use this custom cop that we've developed,
15:25 and that has no dependencies,
15:27 or the dependencies it does have, it just requires them using relative requires,
15:31 so you don't need to do any manual setup yourself.
15:34 So I found this flow pretty cool,
15:36 and again, this is one of the cops that supports autocorrection,
15:40 so you can just get it to go,
15:42 and I actually had to hear like not save the file,
15:45 because I have autocorrection set up by default,
15:47 and it would just like helpfully move this up to the right place where it wants it to be.
15:52 So for me, this has been a cool, I guess,
15:54 slight aside from Ruby specifically,
15:56 but like a cool thing of trying to improve the developer experience for contributing to HomeBrew.
16:01 So we try and push things, like first it was kind of pushing as much as possible into CI,
16:05 and then now with stuff like this we're trying to push stuff that we can into developers' local environment,
16:11 and even into their text editor to mean that the flow that they have is a lot more efficient.
16:16 So, you know, I guess in a bad old days situation,
16:23 like this is a comment that gets made by a human on a PR after you've submitted it,
16:27 and that's a bit annoying.
16:28 In a slight middle ground, this gets kind of, you know, flagged by a CI job which fails when you submit the PR,
16:35 but I guess my view is, is it not nicer still if possible if you can get this right when you're editing the file,
16:41 so you can get that immediate feedback loop and not require like turnaround and CI jobs and all that type of thing.
16:48 So, one of the kind of, I guess, other funny things that we do a little bit differently from other RubyLand stuff is,
16:56 despite being written in Ruby, we do not provide any sort of Ruby API.
16:59 If you are writing some gem or Rails application or whatever it may be and you want to interface with HomeBrew,
17:05 then there is no like direct Ruby way of doing that, and that's somewhat intentional.
17:10 Unfortunately, as a result of doing that, you do get projects.
17:14 I'm allowed to shame this one, I feel, because it was A, made by my employer,
17:19 and B, like I helped to kind of tear down this part of the project and then eventually the project itself,
17:25 but yet historically we've had projects who just, well, they don't provide a Ruby API,
17:31 so we'll just like monkey patch their classes and then call HomeBrew and slightly modify the behaviour to make it do what we want to do.
17:39 Put your hand up if you think this seems like a really good idea for sustainable software development.
17:43 Hopefully, for the benefit of the tape, no hands were raised.
17:48 So, yeah, please don't do this.
17:50 But I will have to admit, I think I've done vaguely similar things to some Ruby libraries in the past,
17:56 so I'm not completely innocent.
17:57 But instead, our API for HomeBrew is meant to be the command line interface.
18:01 So we treat the commands that generate output that looks like it is computer readable.
18:10 We generally try and treat that as a stable API.
18:13 And even when commands behave in a way that is slightly annoying or opaque perhaps in their output,
18:19 we try and respect that such that we don't break functionality for other people in the future.
18:24 For example, when you run BrewSearch, if you have ever been inclined to run that without any arguments,
18:30 then that will print out the list of formula that are currently installed on your machine.
18:34 Sorry, not installed on your machine.
18:35 They're currently available on your machine.
18:37 That seems like a weird side effect behaviour, but there are enough people's scripts on the internet,
18:41 because I go and look for such things because I am, I don't know, bored like that,
18:47 that we don't want to break that.
18:50 So periodically, someone will try and refactor that functionality,
18:54 and they're told, no, actually, we're going to leave this how it is,
18:56 because this is essentially a public API at this point.
19:00 But some of the nicer APIs, I guess, we have instead are things like you can ask BrewInfo,
19:04 which gets you your effectively machine readable version of the formula file,
19:10 if you pass the JSON argument.
19:12 So you can get it to spit out all this stuff,
19:14 and then you can use a tool like JQ,
19:16 with a nice little, like, command line JSON thing,
19:18 or if you're writing Ruby yourself,
19:20 I guess your API to homebrew is now shell out to brew info wget/json,
19:25 and then pass that to JSON parse,
19:27 or whatever your JSON library of choice is,
19:29 and there you go, you have a nice Ruby hash representing,
19:32 like, the various data for a homebrew formula.
19:35 And that also will represent, you can't quite see here,
19:38 but that will represent,
19:39 like, the stuff that you have installed on your system as well.
19:42 Um, if you don't have access to homebrew,

Mike McQuaid

Building Homebrew in Ruby