Hey geekysuperstar305 (or whatever your reddit username is) :),
So earlier yesterday, it was brought to public attention that an npm package used by major decentralised web applications (dApps) had been compromised. This package happened to be @ledgerhq/connect-kit which is maintained by ledger - the most secure hardware wallet provider. Ironic right?
Well - Not really, because the compromised package has completely nothing to do with ledger's secure hardware wallets.
Notwithstanding, a few people still panicked and were very worried about the safety of their hard-earned magic internet money sitting in hardware devices completely disconnected from the internet.
There were lots of speculations as to how this happened and rumours suggested that a hacker got access to ledger's npm keys by exploiting a Github workflow they triggered via a pull request and then successfully published malicious code that caused the package to open a secondary window when a user tries to connect their wallet.
I found the whole thing quite puzzling as I kept wondering if Github workflows were poorly engineered or there's more to this event and eventually, I decided I had to see for myself.
In order to recreate the exploit, I made use of an existing open-source repository and it turns out:
1. A pull request can edit a workflow and the new workflow will be run immediately the pull request is made.
The screenshot below shows the content of a pipeline.yml file in the repository I was testing on. This pipeline is configured to run on every push to the main branch and it deploys the serverless application to AWS.
I created a new branch on the repository and then modified this file configuring it to instead run on every pull request from any branch at all. The screenshot below shows the new state of the file.
In addition to these modifications, I also echoed the AWS_REGION Github secret present on the repository so I'm guessing, I should see the value printed in the workflow logs after the action is successful.
I opened a pull request to the main branch with these new changes and just as I anticipated, the modified workflow ran successfully even when the pull request hadn't been reviewed, approved or merged.
In the screenshot above, you'll see that the workflows triggered by my pull request were run successfully. At this point, you may be wondering why there are two workflow runs here. I'll get to that part in the next paragraph but now let's focus on the bottom one labeled "test: modified Github workflows".
After the run was successful, I checked the logs and the echoed AWS_REGION Github secret was concealed as shown in the shot below. This was anticipated as I had read a similar article demonstrating this exact kind of exploit before carrying out my own experiment.
If the keys were concealed, how then was the exploiter able to grab hold of it? This question led to me second finding:
2. It is possible to expose github secrets by making curl requests to third party apis inside the newly edited workflow.
Yeah, you read that right! I edited the workflow again to include a request to a a request-bin I created on pipedream as shown in the screenshot below.
This explains the second workflow run in the previous screenshot showing that the workflows were successful.
So, on pipedreams's requestbin, I was able to see in broad daylight, the supposed Github "secret" but then, it still didn't make any sense as I thought - So anyone could just make a pull request to any open-source project and grab the keys just like that?
It turns out, there are settings on Github that covers this vulnerability. These can be found on the /settings/actions page of the repository and they include:
- Actions permissions setting: This setting allows all actions to be used by default regardless of who authored them so it may be ideal to update accordingly.
- Fork pull-request workflows from outside collaborators setting: This setting allows you to defined which subset of outside collaborators will require approval to run workflows on their pull requests. I think this is a very important one as the wrong option here can leave your repository and open for exploits like this one.
Also, in this part of Github's documentation, they specifically state that workflows from forks do not have access to sensitive data such as secrets. This means that the sole reason my little experiment was successful may have been because I have full write access to the repository already.
Considering all of these, if the rumours are true and ledger's npm's secrets were actually leaked, it only points to a few possibilities.
- CASE #1: The package maintainers didn't take note of these settings.
- CASE #2: The exploiter somehow obtained full write access to the repository.
- CASE #3: The exploiter had been or currently is a previous contributor on the project.
- CASE #4: It was an inside Job.
Looking closely at all of these deductions, I personally think CASE #3 is the most likely cause of the exploit as an approval may not be required for a previous contributor to trigger workflows on the repository.
I've come across several tweets on my timeline stating that a former employee's account was compromised which possibly proves that CASE #3 is indeed the cause of the exploit but then I'm yet to read the official statement from ledger so I'm not sure if they admitted what the cause of the exploit was.
Anyways, I can now get back to manipulating zeros and ones so that I don't starve to death and playing call of duty in between breaks as I have managed to convince myself that I now have a feint idea of what went wrong and how myself (an imposter) as well as real open-source maintainers can avoid this problem in the future.