Macaroons for Authorization with Neil Madden

Damian Schenkelman: Welcome to Authorization in Software, the podcast that explores everything you need to know about authorization. I'm your host, Damian Schenkelman, and in each episode we'll dive deep into authorization with industry experts as they share their experiences and insights with you. If you're a software developer or just someone that's interested in the world of authorization in software in general, you are in the right place. Let's get started. Welcome to another episode of Authorization in Software, where we'll dive deep into tooling standards and best practices for software authorization. My name is Damian Schenkelman, I'm chatting about authorization on my program with Neil Madden, founder at Illuminated Security and author of API Security in Action. Hey Neil, it's great to have you here.

Neil Madden: Hi Damian, it's great to be here.

Damian Schenkelman: Could you give our listeners a brief overview of your background and what are you up to today?

Neil Madden: Yeah, of course. So as you said, I'm Neil Madden and I'm currently running a startup called Illuminated Security. We're providing application security, training on things like JSON Web Tokens, things like that and a bit of consultancy. Before that though, I was the security architect for ForgeRock, where I worked for a bit over nine years off and on. I'm also the author of the book, API Security in Action, which was published by Manning a few years ago. And in my spare time I'm also a bit of a security researcher, take a look for vulnerabilities in software. So some of your listeners may know me from a major vulnerability I found in Java's Elliptic Curve signature software last year, which is called Psychic Signatures. I'm also quite a long time regular contributor to the OAuth working group at the IETF. So that helped shape some of the more recent standards that have come out of that group and pretty active there.

Damian Schenkelman: Sweet. It seems you've been thinking about cryptography and how to use it to secure APIs for quite a while, which is a big reason that, hey, I'm really excited about having you here. I've been thinking about doing a macaroons episode since I started the podcast and when I started thinking about it, I was like, who can I invite then? And your name came up in conversations with other folks, I read your post, I was like, yeah, Neil's the person to have on here. Could you maybe share with folks what a macaroon is and why we're talking about it in the context of authorization?

Neil Madden: Yeah, of course. So macaroons were invented by Google, they produced a research paper, I think almost 10 years ago now, where they described this new approach to authorization tokens. So they're really, in some ways they're similar to JSON Web Token or jots, but they're much more constrained. They only have one algorithm they support and it's a very simple algorithm, but they have some interesting new properties that jots don't have. So in particular you can add these things called caveats to a token after it's been issued. So with a jot, it issues a token and then it's fixed for the life of that token. Whereas with a macaroon, after it's been issued, anyone can add these things called caveats to it, which restrict how that token can be used. So there's this nice thing you can do where you can change the token after it's been issued.

Damian Schenkelman: Nice. So we introduced a few concepts there, we talked about JSON Web Tokens and we also talked about algorithms. Can we maybe share what's a JSON Web Token? People might not be familiar with it. And also, what's an algorithm? When we think about the JSON Web Token, what's the algorithm? What is it used for?

Neil Madden: So hopefully people are familiar with cookies and things like that. So when you go and log into a website, you get a cookie set in your web browser, which remembers who you are. And the traditional way those were done, was you logged in and then details about the user was stored in some backend database and some random ID was generated. That then became the cookie and then every time you sent the cookie back, you'd go and look it up in the database and find the information. And so JSON Web Tokens and things like that are a way to do that without having the database basically. So you can encode all of the information into JSON or some other format like that, and then you sign it cryptographically with a secret key, to protect it against tempering, for threats and things like that. And then that encoded token becomes your cookie, or your access token, or whatever else you're using for authorization and then you don't need the database. And so when a token is presented, you can verify the signature, check out has it been tampered with and then extract the details from the token. So JSON Web Tokens support lots of different algorithms. So the simplest one they support is something called a HMAC, which stands for hash- based message authentication code, which uses-

Damian Schenkelman: We're talking about algorithms for signing the token, you mean?

Neil Madden: Right. So you can sign it either with a thing called HMAC, which as I said is a message authentication code where you have a secret key and both the person signing, issuing the token and the person verifying it have to have access to that shared secret key. But then JSON Web Tokens also support public key algorithms where there's a private key that signs the token and then there's a public key which could only verify the tokens and then you can distribute the public keys to the PI servers or whatever and they can verify the tokens without being able to it. So macaroons just use HMAC basically, so it's always based on a shared secret. So this is pro to con for macaroons, but that's the one algorithm they support. And so the issue and the verifier of tokens and macaroons have to have the shared secret key. On the plus side, you get these nice properties of being able to append these caveats and HMAC is really fast, it's a lot faster than most public key SIGTRAN and things.

Damian Schenkelman: So to recap, and let me know if I got this right. Back in the day we started soaring some opaque large string in a cookie and that was a reference to something in the database, that represented data about your authentication on your session. At some point we said, " Hey, can we make this stateless? Stateless is easier to scale, because you don't require state in the database. We can just horizontally scale adding more servers." Yes, we can use cryptography instead. So we started storing JSON sign and again, that could be signed with a typical secret share key or it could be signed using a public or private key cryptography. At some point and then you said, " Okay, that's a JSON Web Token." The thing about the JSON Web Token is that once it's been signed by what we refer to typically as a signer or the issuer, it doesn't change and this is where macaroons come in. You can add these things called caveats that allow you to change the macaroon and in exchange for that, again, you need to rely on using shared secrets, you can't go with the public key format. What's a caveat and how can I use it? What are the use cases for it?

Neil Madden: So there's a couple of different types of caveats. The simplest ones are what's called first party caveats, and then there's another thing called third party caveats, which we can maybe talk about later on. So first party caveats are, they're just a restriction on how the token is used and they are just in macaroons, they're just a string which describes some condition that has to be true When the token is used. In terms of things like cookies or excess tokens and things like that, that people will be familiar with, you could think of it's like a caveat as being, so for example, the expiry time on the token. When it's get issued, it will have some expiry time, which may be two hours in the future or 24 hours or whatever. And you could add a caveat that says, " Actually now the expiry time is only five seconds in the future." So why is that useful? You might think, well, if I've got this token, why do I want to limit myself to it only expiring a lot sooner than it was to other? Well, the reason for this is when you're going to send that token somewhere where you don't fully trust where it's going, you don't trust the network or you don't trust them sitting on the other side. So if you are about to make a network request to go to access some API, then you've got a cookie or something that is valid for 24 hours. If that cookie gets stolen on that request, then the attacker can use it for 24 hours until it expires, which is quite a long period of time. But what you can do with macaroons is you can create a copy of that token, that cookie, and you can add this caveat saying, " Actually it only expires in five seconds." And then send that version over the network and then if somebody steals it, it's only good for five seconds. And you can add different types of caveats, in the original paper they call these contextual caveats. You are restricting the token based on the context of the request you're just about to do. If you were doing some transaction and you had some unique ID in that request, then you could maybe add that as a caveat and then it's only valid for that one request and things like that. But you've still got the original macaroon without those caveats attached to it. So you can still use it for 24 hours or whatever, and every time you make a request, you create a copy of it and you answer more restrictions related to the request you were just about to make. And then that means that if those copies get stolen, either on the network, or because of some insecure coding on the server, or leaking in server logs, or all the other ways that these things can happen. Then what's been stolen is a much more restricted token than the one you originally issued.

Damian Schenkelman: It seems using macaroons and some of the use cases becomes interesting and you take advantage of them and we have different parties in the system and the one holding, I would say an original macaroon doesn't have. I'm going to use quotes, although people cannot see them, high level of trust of other parties in the system. But we use you and hey, who can issue it and who can add caveats to a macaroon, maybe we can go a step before and think about an example. So what's the story of a macaroon? How does a macaroon get issued, and what's a typical party that would issue it and who would it issue it to?

Neil Madden: So there's lots of different ways and scenarios. So they could be used in the same way that JSON Web Tokens are used for all kinds of different situations. So to issue the macaroon, you just need to have this secret key that you use to create it. And then they have this identifier which is just an arbitrary string that you can put stuff in. So that could be something like you could still have a traditional database token and have my an that's opaque in the first step, or you could put in some structured JSON or something like a jot claim set, and then you sign it with your secret key using HMAC and then you issue it. And so if you think about the examples, so you can use them as a better cookie, which was how Google originally pitched them. So you go inside your website and then your website could use as a macaroon, as a cookie and send it to the server, which you're then going to send it back. You'd have to have some JavaScript on the web client, which is then looking and appending these caveats to them. It'd be nice to think maybe in the future Google could change Chrome to automatically add caveats when cookies were sent in, but I don't see them doing that anytime soon. And then there's also, so one of the things we were looking at, ForgeRock was using it within the OAuth system. So having access tokens issued by an authorization server as macaroons, which would then be verified by resource service, some APIs and so there was a couple of ways we were looking at doing. So as I said before, because these are based on shared secret keys, you can either trust all your APIs and give them access to that secret key to verify stuff. But then they also then have the capability to mint their own access tokens, which is more trust than you typically want to give them. Particularly then, if any of those servers gets compromised and that key gets compromised, then you you've lost the keys to your whole authorization kingdom really. So a different way you can do it is to issue these macaroons as tokens, but then still have your API servers call back into authorization server to validate using something like token introspection, which is the standard OAuth API for validating an access token. So that was typically used with database backed stateful tokens, but you can use it with macaroons as well. I mean, obviously then you've got the latency of these network overheads when you want to validate tokens, but you get things like ease of revoking tokens and you can still horizontally scale your authorization server or inaudible.

Damian Schenkelman: That's neat. There's a bunch of concepts here and I'm going to try to unpack them and when dig deep in with a few questions. So it seems there are a few scenarios where you might start issuing a macaroon. You talked about two, you said you have an app or a server, someone starts a session with you, you issue a macaroon to the client and then the client sends it back to you with again, when they're using it as part as the cookie, as part of an API call to the same server. And because you issued it, you naturally can sign these, which means you have the secret to verify them. Now that that's one scenario. In that case, so let's talk about that one in particular. Who is a party that gets the macaroon? Who would add caveats? Where do you see, okay, I'm getting a macaroon from, let's say a website that has authentication baked in, they are not delegating with OAuth or anything? Who would get that macaroon and essentially add a caveat to it, attenuate it to share it with another party and what would that party or what could that party be?

Neil Madden: So there's a bunch of different things here. So one example is just the client is something like a web app and it's JavaScript running in your browser, or it could be something like a mobile app and things like that. And it's taking this token and then it's just doing what we said before, so it's adding these contextual caveats before it makes API calls. So that as a pure security mechanism, just if that token gets intercepted somehow or leaks somehow in that request, what gets stolen is less of an impact than if the original is stolen.

Damian Schenkelman: So in that case it essentially it's attenuating or rather in caveat and then it sets for itself, so it has the updated macaroon with some restriction and then that's what sent over the wire. And what you're trying to limit there is, if someone can steal that over the wire, what they steal is no longer maybe going to be valuable in five seconds or next time. Is that one use case?

Neil Madden: Yeah, that's one use case. So it keeps the unrestricted version to itself and then it can send this more restricted version over the network. Because you might as well, a client might get an access token which is valid to access a whole bunch of different APIs, which are sitting on different servers and maybe run by different people and so on. And so if one of those servers is compromised or something, then when it gets your access token, it can use that access token to access all of the other servers as well. Whereas one of the things you can do with caveats is you can add an audience restriction to a token after the fact. So the original token might be valid for these five different servers, but before I call this wand, I will add an audience restriction that makes it just valid for them. So for instance, if you look at Google Cloud for an example, they have, you get an access token from them and you can access hundreds of different APIs in the cloud with this token. And so you might have a token that can access all kinds of backend services there and that's useful for your app, because it needs to access these at different times. But when you make the actual calls, you can append this caveat saying, " Right now on this call, that's only valid to this one API." And then if somebody steals it, they can only access that one API with it, they can't access inaudible.

Damian Schenkelman: In that case, you're making the client an issuer, let's say the server, but let's say the issuer of the macaroon on the client, it's a less chatty relationship. They just get the macaroon once and the client attenuates it when calling the API and server, rather than having to go get, I would say a more restrictive version each time they want to call one particular API. So that you're gaining security by putting the owners on the client to restrict things, while at the same time reducing load on your servers in terms of not having to issue this attenuated specific versions of the tokens.

Neil Madden: Yeah, exactly. There's also some things you can do when we've some more advanced cases, when you start looking at these third party caveats, which we haven't talked about yet.

Damian Schenkelman: Before we get into third party caveats, which I know are a thing in and of themselves, I wanted to chat more about the OAuth case. You mentioned, we explored issuing an access token that is a macaroon and you talked about, well, but you get into the problem with symmetric keys. Maybe we can talk about typical parties in the macaroon system and typical parties in the OAuth system, how could these be the same and what does it mean? You also hinted at some of these things like, " Hey, if I use my macaroons, I need to share the secret between the authorization server and the resource server." How does that compare, for example, with the JSON Web Tokens and is that even possible? What are the trails here?

Neil Madden: So as you said with this shared secret thing, I mean, when people use JSON Web Tokens and OAuth, what they normally want to do is they have the authorization server has the private key, it signs the token. And then they distribute the public key to all of their API servers, their resource servers and they can then validate those tokens without having to coordinate or talk to the authorization server at all. With macaroons, at least with the version of macaroons that's been used and deployed everywhere, you can't really do that pattern, because you have to share this secret key. So you'd be putting way too much trust in your resource servers then really. And so a typical way you would use them is you'd use token introspection, which is an OAuth standard, which is why this is part of the reason why I think macaroons and OAuth work quite nicely together. Which is that macaroons lack, there's no standard for macaroons and there's no standing for these interactions and things. Whereas OAuth provides this nice framework for how these interactions happen, but it doesn't specify what the tokens look like. So you can plug in and get the best for both with these. So what you can do is you can have your secret key is just in your authorization server and the clients talk to it to get an access token and that will be a macaroon. And then when your resource servers want to validate that access token, they call the token introspection endpoint, which is a standard OAuth API, to validate an access token. So they call into it and say, " Hey, here's this access token and I want to validate it." And one of the nice things we were able to do at ForgeRock when we implemented this, is that the response to that token introspection API is a JSON document that says, " Is this access token valid? When does it expire? What's the audience that it's intended for? What's the scopes and things like that?" And with macaroons, what we allowed is people to add caveats that can restrict any of those things, so they can restrict the expire time, restrict the scope and all those things. And the authorization server would look at this macaroon that came in, it would validate the scope, the signature, then it would look at all of these caveats and it would automatically incorporate them into the introspection response. So as far as the resource server is concerned, it just looks like an ordinary introspection response for any other access token. But if you've appended a caveat that says, " This expires in five seconds." Then the expiry time on the response will reflect that. If you've appended, if the original access token had five scopes that it was validated for and you appended a caveat that reduced that to two, then the token introspection response will just show those two scopes as being the scopes of the access to.

Damian Schenkelman: That makes sense. I can imagine there are some trade- offs there, particularly if we think about how we started the conversation. We said we went from opaque cookies with opaque strings that were references to a database, to we want to use crypto so that we can scale this. And if we go with an approach like this, the resource server has to call the authorization server to do introspection, which means that there's added latency in that call. And depending on where your resource servers, your APIs are deployed, they might be around the globe, they have to call maybe a centralized location if you have a centralized authorization server, it also means that your authorization server becomes like a point of failure for your API. How do you think about this trade- offs in the design and maybe what other trade-off are there when using such a design?

Neil Madden: That's exactly, those are all really good points that people need to consider. So all of those things are true. So there is some extra latency authorization server, there is then becomes a single query failure and things like that. Where I think it's different from the database approach is that you can still horizontally scale your authorization server. So a database typically is really hard to horizontally scale anything that has state that you have to synchronize and look it up. Whereas here you can distribute that secret key to all of your cluster of authorization servers and you can scale those. So even though you're calling back into this authorization server, all it has to do is validate the signature, so it doesn't have to look it up. So there is still the opportunity for more horizontal scalability and in return for that you get some nice features. So it's easier generally to check in whether the token is being revoked and things like that. There's also something I think which is not talked about often in Oauth, is the traditional software engineering idea of encapsulation, which is the idea that you should hide the details of things that you might want to change later. One of the things you might want to change later, is what kind of access inaudible, what kind of format. So if you say access tokens are jots and all of my API servers know that and they will validate them, then it's really hard later on to ever change to something other than jots. Whereas if you are calling back to the authorization server to introspect the tokens, then what format those tokens are is encapsulated by the authorization server and is hidden from those things. And so you can start with a database backed token, change it to something like a macaroon. Maybe later on you decide actually macaroons were wrong, I would change it to jots and your API servers don't have to change, because the authorization server is handling it. So there's these trade- offs and the trade- off for that is that you have this added latency and potentially single point failure and those kinds of things.

Damian Schenkelman: I really like how you not only added to the trade- offs, but also explain this notion of encapsulation. And at the end of the day, I think you have to make choices that are informed for what you think your application needs. For example, if you think about, if you're doing authorization for a B2E application. Well, your employees are rightly going to continue using whatever you're doing, because first of all they're stuck there and also, it's not latency sensitive. At the same time, if you think about maybe e- commerce, where it's well- known that latency affects the conversion on how much people buy, that you might go with another approach and you trade- off encapsulation, but you gain performance. That's very interesting. I want to now double it down on some of the capabilities of macaroons, because again, we've talked about the caveats and the first party, we've talked about how you can use them for verification, particularly in the OAuth flow. But you mentioned this thing about third party caveats and we talked about it a couple of times, and now I think it's time to open up that box. What is a third party caveat and how does it come to the table?

Neil Madden: So third party caveats are really amazing and also a bit scary at the same time. So with first party caveats, we had it as they're just a condition, they're just some string that you have to check. With a third party caveat, the idea is that you are adding a condition to a token that requires the client to go to some third party service and get a proof that it's set aside some condition. So it's not something that the resource server or the verifier can check themselves locally, it requires some other service to do. So a classic example that's used a lot, which I think helps illustrate it, but perhaps is not that real world, is the idea of proving that you're over 18 or over 21 or something. That you're going to use this macaroon to go and buy some alcohol from an online store and that wants some proof that you are old enough to buy this based on the local laws, whatever. And so the idea is it can issue this token and it has this third party caveat that points at some government age verification service. And so you as a client then have to go to this service and get this proof, which is known as a discharge macaroon that proves that you are over 18, or over 21, or whatever it is. And then you present both your original macaroon and this discharge macaroon when you want to buy your alcohol and it can tie the two together. There's some clever cryptography that's used for that. So another use case where I was working on at ForgeRock, which this was when there was a lot of talk around open banking and things and PSD2 and these financial APIs, and wanted to fine- grained authorization of transactions. And so there was a demo I came up with which was around that kind of thing. So the idea is that you would have, you'd install a banking app on your phone and then you'd do a normal OAuth 2 flow for that app to say, " Yeah. I want to be able to issue transactions from your bank account and stuff." And so you'd approve that as a normal L2 flow and you'd get a macaroon access token. But you don't want to just let that app just initiate whatever transactions it wants, you want to be able to actually approve everything it does, individually. And so you can attach a third party caveat to that access token, that requires it to go to a separate transaction authorization service every time it wants to perform a transaction. And that transaction authorization service would go and actually communicate with the user, say, " This app is trying to transfer$ 2, 000 whatever to this thing, is that okay?" You'd approve it on your phone or whatever. And then once that was all approved, this transaction authorization server would issue this discharge macaroon, which authorizes just that one specific transaction. And then the app can then present these two tokens together, to then go and actually to the bank, to actually go and initiate this transaction. And so you get this long- lived access token that authorizes the app to even start initiating a transaction, to even ask for a transaction, and then you get these short- lived discharge tokens that approve the individual ones.

Damian Schenkelman: I get it. So the client seems to have an incomplete puzzle that the server will then verify and it needs to figure out how to put this password together. You mentioned a couple of cases, it'd be covered in age verification service, the financial transaction approval. How does the client put the parcel together?

Neil Madden: So the way these third party caveats work is the caveat, whoever creates the caveat adds it to the original macaroon. They create a new secret key and that secret key gets encrypted and added into the caveat, in a way that the verifier of that macaroon will be able to recover it, but nobody else can. And then when the client sees this caveat on the token, it knows it's going to have to go and fulfill this requirement, but it can't access that secret key. So what it has to do, there's a location hint which tells it where to go to go and get this track route and there's no actual standard in macaroons for how this happens. So there's a bit of some missing gaps here and you have to invent your own conventions for this at the moment. But basically the client will go to this URL and it'll present this caveat to the URL and that third party service of that and that URL will then check, does this client satisfy this caveat and it will be able to understand what the caveat is. That service will also be able, well, there's a couple ways this happened, but often it will be able to decrypt something to get this secret key. So if it's satisfied the client satisfies the condition that's in that caveat, then it can recover that secret key and it uses that secret key to create a new macaroon which is called the discharge macaroon. And then it issues that discharge macaroon to the client and that's its proof that these things happen. And then the client then presents its original macaroon and this discharge macaroon to the API, which then verifies it and as part of its verification and from the original macaroon, it will decrypt this secret key and it can go, " Hey, look, I've got this discharge macaroon that corresponds to this and I'll use the secret key to verify that discharge macaroon." I mean, that's mind- blowing itself. Where it gets even more mind- blowing is that that discharge macaroon is itself a macaroon and you can add more caveats to that macaroon, including new third party caveats. So in general you might have this tree of caveats that have to be verified and things.

Damian Schenkelman: You might end up, as you say, with a tree, with a very long chain of things to verify and ultimately it seems that again, there is no end to it and you could keep adding macaroons to it and so on. You also mentioned that there is no standard on how to agree on a location and essentially how to get a discharge macaroon for a third party macaroon to be able to fulfill the caveat, to show that you're proving that it's successful. What other things do you need to informally agree on a system to make macaroons work?

Neil Madden: So this is in my opinion, the biggest weakness with macaroons at the moment is that when you go and look at something like JSON Web Tokens, there's an RFC that defines JSON Web Tokens and then there's four other RFCs that it builds on, that define all the cryptographic algorithms and things like that. There is nothing like that for macaroons. There is the original research paper and then there was a defacto standard implementation that was done, called NIB macaroons, that's been ported to different languages, but a lot of these details are just left blank. So things like even for first party caveats, they're just strings and what those strings are is not specified. So it's not like with jot claims where you have an EXP claim, that sets an expiry time and things like that, there's nothing like that. So that all has to be agreed. So in ForgeRock, when we did our implementation for OAuth, we based those things on existing things around jots and so on. So if you want to restrict the expiry time of a ForgeRock access token macaroon, you add a caveat which is a little JSON object with an EXP thing with an expiry time in it. And likewise, or if you want to AUD, if you want to restrict the audience scope, if you want to restrict the scope, things like that. So we based it on that as a standard, but that makes us non- standard with other macaroon implementations which use other ways of doing this. And the same then with these third party caveats, there's a format for how you encode the caveat, and you can have this little encrypted key and some little identifier that gets given to the third party service. And you can put a location hint in there which tells the client where it has to go, but what it does when it gets to that URL is completely unspecified. Does it do a post? No. Does it send something to JSON? How does it communicate this caveat to it? So all of those details and even at ForgeRock, we never really got round to formalizing what those details would look like. Third party caveats left as a work in progress I think, I don't know whether they've done more work on them since I left. So there's a lot of gray areas here, where at the moment they work better in closed ecosystems where you can control all of the parties and you can nail down these details yourself. I'd like to think if macaroons catch on, that there'd be more standardization effort around this and they'd mature and these things would get sorted out. Let's hope for the future, but at the moment you need to be able to do these things.

Damian Schenkelman: What implementations of macaroons in the while are you aware of?

Neil Madden: So there's a few I'm aware of, there's not that many at the moment. Again, this is a flux, the more maturity level that they're at the moment. So Google came up with them and I'm not aware of Google actually using it themselves anywhere. So at ForgeRock, obviously we added them and I know some customers were looking at them. By the time I left I don't think any customers had gone live with a macaroon implementation, but certainly some were interested. There's a few things I'm aware of, the Lightning Network, which does wait transactions on top of Bitcoin, that uses macaroons and it uses third party caveats for micropayments. So you can have an invoice as a third party caveat attached to some token and then when you pay the invoice, you get the discharge macaroon that cashes it out and then you can use that. So that's a nice thing. I know Ubuntu Linux, they have their Ubuntu One service, which is their cloud service. That's their single sign on service behind there was all built with macaroons in the background. You won't find any mention of it in the documentation and things like that, but if you go and look in Stack Overflow for developers saying, " Oh, I got this error message from this service." You can see mention of macaroons all over the stack traces and things. So that's all built on macaroons as far as I understand or at least was. There's a version of macaroons, a newer version called Biscuit, which we haven't really talked about yet. So some people got disappointed with these restrictions of macaroons to only be the shared secret and they came up with this thing called Biscuit, which is a public key version of macaroons, it uses public key signatures. It's a bit more expensive to do things, public key crypto is more expensive than things and there's some different trade- offs. They've done some interesting things around there. They define a really interesting format for caveats based on this thing called data log, which is if anyone's ever done prologue logic programming at university and things like that. So data log is a simplified version of prologue, so you can encode some really complicated and interesting authorization rules in Biscuit. So there's Biscuit then and there's a company called Clever Cloud, who were involved in the development of Biscuit and they use Biscuit then in a lot of their services as well, for their authorization of their cloud services. Those are the main things I think I'm aware of, there may be others that I don't know about.

Damian Schenkelman: I think Biscuit opens up a whole new path to go down towards and that might be a very, very good topic for a future episode. The last thing I want to ask is the flip side of that question. When should a developer or a team use macaroons? What problems should scream macaroons at them and what production recommendations would you give them to basically get a system working and not have issues?

Neil Madden: So I think for me, one of the things I think that that works really well for macaroons is when you want to harden security for a system, but you don't want to deploy something more complicated. So by that I mean there's a lot of work now on, looking at things going beyond bearer tokens too. So all the stuff I'm talking about so far, they're bearer tokens, they're like cash, if you have them, you can use it and so if somebody steals it, they can use it as well and so there's a bunch of standards around this. So tying access to things like TLS certificates or is this standard DPoP now, dynamic proof- of- possession and things like this. Which you try to harden up these tokens and add more security by tying them to some private key to them. So you can't just use the token, you also have to have this private key that you sign something, or you set up a TLS session, or something like that. But the problem with these solutions is that they're really hard to deploy after the fact, because you have to upgrade everything at once. Your authorization server has to issue these bound access tokens, which is bound to these private keys. Your clients have to know how to request these access tokens, how to manage the proper keys, how to then prove possession of the private key when they're doing stuff and all your resource servers need to know how to verify those things. If you're using the token introspection we were talking about before, where your servers are calling back into the authorization server, then you can switch to something like macaroons and it's transparent to most of your software. So your authorization server can start issuing access token macaroons and nothing else needs to change initially. So your resource servers that are validating the tokens, they're still just only token introspection, it just looks like another opaque access token to them. The clients can treat it like an opaque access token to begin with, but then you can harden things slowly. So you can change your clients one at a time to start adding contextual caveats before, like I'm saying before they make requests. And so you get some of the benefits then that these are hardened tokens, that if they're stolen, they're harder to use, they're induced scope and things like that. So that's one of the nice things I think using them. And then some of the things around these fine- grained authorization things we talked about before with caveats, things like that. In terms of production-

Damian Schenkelman: Sorry to interrupt. How should a dev that maybe's not an expert in identity and security think about this? Because on the one hand, macaroons offer these capabilities in an implementation manner, that's probably simpler. But if you look at standards, if you look at the Open ID working group and so on, you get guidance for the DPoP stuff. So there is also some things to balance there maybe.

Neil Madden: Yeah, definitely. I think that there's a trade- off. I think at the moment for where macaroons are on their level of maturity, I would recommend that they be used only in a closed ecosystem, internal deployments at the moment. Hopefully over time we'd see them mature more and some of these standardization gaps filled in, and then they might be more suitable for deployments external and facing deployments where you have more different actors involved and different parties involved. So at the moment, I would definitely recommend with the level of maturity the technology is at that you try and deploy them. If you're looking at deploying them in a environment where you can control all the moving parts, to make it a bit easier for yourself. I think there's also, in terms of technical things that there are some recommendations. So the fact that it's all based on this shared secret key, means that these keys have to be really protected and so you need to look at key storage and key rotation. That are really important things to look at and making sure you're not using the same key for everything in your entire state, but you are segregating keys. And I use this one key for signing macaroons for these APIs, I use another one for these other ones. Also, there's a really great talk from a few years ago, one of the few real in the trenches, production deployments of macaroons. It's by inaudible, hopefully I pronounced the name correctly, a few years ago. And that was on, it's a failure case really of deploying macaroons, but there's some really good lessons learned in there.

Damian Schenkelman: Yeah, I saw that one and we're going to add the review to the show notes.

Neil Madden: So one of the things they did there was they made a mistake with third party caveats basically, and this is really instructive I think. In that they had this production or service that was all designed to handle a lot of scale and handle their traffic for their systems. But then they added a third party caveat that required clients to go to what was basically a toy, I don't know if it was a toy, but it was a small scale Ruby on Rails app, that just hadn't been sized appropriately to deal with the amount of traffic it was then going to do. And what you've basically done then is create a production dependency of your auth service on this tiny little app that is not scaled appropriately for it, and so every time that app went down, you basically took out auth for everything really.

Damian Schenkelman: I think it was exactly what you said, that they were validating macaroons on their API, but they had a third party caveat on the dashboard and the reliability capabilities of those components were fairly different, which is common for almost all products and that was a big issue that they had.

Neil Madden: Yeah. So I think generally my advice would be, I feel first party caveat to a lot easier to understand and a lot easier to deploy and read about. Third party caveats are like you read about them and you go, " Wow, these are amazing." And you think of all these ideas, things I can use them for, but actually they're also a good way to shoot yourself in the foot. So they maybe should be something that you get some experience with the technology first, then gradually think of introducing.

Damian Schenkelman: The typical, with great power comes great responsibility.

Neil Madden: Yeah.

Damian Schenkelman: Again, it's been amazing to chat with you about these topics, you know a lot and you can clearly explain them in detail for layman terms. So I really appreciate it. I'm really happy we got to chat about this and maybe in the future we can do another episode with Biscuits.

Neil Madden: Yeah. Well, there's also, I'll try, but I'm going to be presenting something hopefully at the OAuth Security Workshop in London, which is my own take on some of these things. So look out for that.

Damian Schenkelman: Yeah, definitely. And we can add that to the show notes once that becomes public and we make this episode available.

Neil Madden: Great.

Damian Schenkelman: It is great having you, Neil. I really appreciate you taking the time for this and hopefully every listener here had a good time learning about macaroons and how you can use them for authorization. Thanks a lot.

Neil Madden: Brilliant. It's been great being on. Thank you for having me.

Damian Schenkelman: That's it for today's episode of Authorization in Software. Thanks for tuning in and listening to us. If you enjoy the show, be sure to subscribe to the podcast on your preferred platform so you'll never miss an episode. And if you have any feedback or suggestions for future episodes, feel free to reach out to us on social media, we love hearing from our listeners. Keep building secure software and we'll catch you on the next episode of Authorization in Software.