At Unchained, we are proud to work with a variety of amazing hardware wallet manufacturers to design self-custody products that minimize trust and eliminate single points of failure from the bitcoin custody equation.
Working with multiple vendors that all interact differently in different browsers, with different operating systems, and different design decisions leads to interesting and unique experiences that I hope to share with the intention of helping you appreciate the importance of physically securing the seed phrase(s) to your bitcoin wallet.
Hold your own keys and use a hardware wallet
Before I provide some accounts of instances where a hardware wallet could have led to a loss of funds if the user did not keep a seed phrase backup, there are two things to keep in mind:
- You should still hold your own keys: I’m not trying to scare anyone away from self-custody or multisig, which is the most secure way to store bitcoin.
- You should still use a hardware wallet: it’s normal for developing technologies to have the occasional errors, and in all the instances described below, the client’s funds remained secure. A hardware wallet is still the best way to conveniently and securely store your bitcoin keys offline. In particular, the hardware wallets mentioned in this article—Trezor, Coldcard, and Ledger—are leaders in the field of bitcoin security.
Rather, I’d like to document some case studies for prospective clients, hardware wallet manufacturers, and multisig wallet coordinator developers about the different things that can—and do—go wrong with software as it evolves.
I want to make the definitive case that you must secure seed phrases physically to be truly secure and trust-minimized with bitcoin.
1. Trezor API update and key path lockdowns
On Thursday, January 28 2021, I received a dreaded client email: “I’m getting an error message when trying to sign a transaction with my Trezor.”
The client had not recently updated their Trezor firmware or the Bridge (which allows Trezor to communicate with third parties like Unchained), Unchained had not pushed any code updates, and signing transactions with Caravan, our trust minimized external recovery tool, was working flawlessly with Trezor. What could it be?
The Unchained Capital engineering team was able to identify and fix the problem within a few hours. It was a trivial parameter in the Trezor API (which allows third-party services to communicate over the browser with the Trezor device) that was changed from “optional” to “required,” which was not publicly disclosed prior to release.
In this scenario, because Unchained has Caravan, our open-source external recovery tool, and we recommend safely storing multisig wallet configuration files by default, clients had an option to immediately access their bitcoin and sign with their Trezors.
And since Trezor and Unchained recommend storing seed phrases on paper, if one of our clients needed to transact immediately and Caravan hadn’t worked either, they could have rebuilt the vault and signed with Trezors using Sparrow or Electrum, or loaded the seed phrase(s) onto a Coldcard or Ledger and moved their funds from their Unchained account.
If we had not recommended clients secure their seed phrases and wallet configuration file, our clients would need to wait for Trezor and Unchained Capital engineers to patch the functionality before they could move their bitcoin, even though they were always in control of a majority of their keys. Thanks to hard work from both teams, the crisis was quickly averted and signing transactions with Trezor on our platform is fully functional again.
2. Key path updates
In 2020, the SatoshiLabs team (the team behind Trezor) locked down nonstandard key paths (BIP32 paths) in order to prevent malware from locking client funds with a key located in an obscure location. A BIP32 path can be thought of as directions (or a file location) from a seed phrase to a specific public and private key.
This upgrade was not an issue with singlesig addresses, since singlesig addresses have clearly defined path standards, but it did cause significant trouble with many multisig wallets, which don’t yet have as clearly defined standards.
For example, In January 2021 these BIP32 path restrictions degraded Unchained Capital’s Trezor signing functionality, and in June 2020 Trezor was for a period of time unable to sign Casa and Green Wallet multisig transactions as a result of these path restrictions. Electrum wallet uses the HWI interface to communicate with Trezors (Unchained and Casa communicate with Trezors via the Trezor Bridge) which was also degraded in January as a result of the path restrictions.
Luckily, in the 2020 case, Unchained Capital and Caravan multisig addresses were unaffected, but this was pure luck, as we saw when signing was degraded in the 2021 BIP32 path restrictions, which didn’t necessarily affect other software wallets.
Most importantly, this demonstrates that any hardware wallet team could unilaterally decide to disable specific paths or signing functionality at any time.
Unchained prioritizes eliminating ourselves as a single point of failure with seamless external recovery, encouraging clients to always secure their seed phrases offline. Therefore, when our clients ran into this issue, they were able to load their seed phrases onto an alternative hardware wallet (Coldcard or Ledger) and sign the transactions from their Unchained account.
Not having seed phrases in this scenario would have required the client to wait multiple days and rely on Trezor’s wallet engineers to patch the issue. Or they could have downgraded their firmware using an advanced, technical, and time-consuming workflow.
These issues aren’t exclusive to Trezor by any means, but do highlight the risks associated with putting too much trust in any third party, even a software or hardware wallet provider.
3. Coldcard requires all fingerprints to sign PSBT
In their 3.2.1 firmware update, Coldcard patched an issue that one of our engineers, Justin Moore, discovered. This firmware patch caused Coldcards to reject certain older versions of the multisig configuration files, Unchained Capital’s included.
The result of this patch was that an Unchained Capital client could build a vault with a Coldcard updated to the most recent firmware, but then couldn’t sign vault transactions with the Coldcard until our Coldcard configuration file was updated with the new standard.
This vulnerability was patched same-day by the Unchained Capital engineers, but it could have resulted in a scary client experience where someone would be able to create a multisig vault but would be unable to withdraw funds from their vault or with Caravan for a period of time.
Another unfortunate edge case surfaced after this upgrade was released. Since it was a backward-incompatible update, meaning you couldn’t downgrade, Coldcards that secured multisig wallets built from the same seed phrase with different xpubs (don’t try this at home—we used to do this occasionally for demo and QA multisig wallets) received the error message of death.
By backing up seed phrases, folks affected by this issue could load the seed phrase onto a Trezor or a Ledger and move funds from their Unchained Capital account, Caravan, or Sparrow.
4. Ledger vs Windows 10 and Google Chrome
Early in 2020, Microsoft released a security update for Windows that caused Ledgers to trigger Windows security warnings in the browser during public key exports, key checks, and signing workflows. This happened because the Windows 10 operating system decided that U2F connections were security risks, and Ledger used U2F to connect with third parties through the browser.
Ledger was stuck in a tough position because their flagship device, the Ledger X, was not yet functional with an alternative connection method, called WebUSB. On Windows, the only functional way to use Ledgers for most of the year was to use a Ledger S in Google Chrome. The Ledger X workaround was to drag the windows warning to the side, which would flicker the entire time, and then click through a series of prompts multiple times. It was essentially unusable.
Later in the year, Ledger released Windows WebUSB functionality for the Ledger S and X, but then Google released an update to Chrome that broke the WebUSB functionality. Now Ledger owners couldn’t use their devices at all with Windows or macOS unless they navigated an advanced browser setting in Chrome to roll back the Chrome update. If the Chrome update was rolled back, Ledgers were back to functional, that is until Ledger released an update that fixed the Chrome WebUSB issue, which required all clients that had updated the browser to go back and undo the browser setting.
This example was a critical problem for Ledger users that was introduced by Windows and Google Chrome. That said, for 2020, many of our Ledger clients had to move seed phrases to a different manufacturer. This was trivial for our clients because they could load their seed phrases onto a different hardware wallet and perform a key check without needing to move their bitcoin.
Always secure your seed phrases
Most major wallet and hardware wallet vendors strongly encourage protecting seed phrases, and for good reason. Securing your seed phrases is crucial because it gets you closest to being truly sovereign over your wealth—as trust-minimized as possible. Following best practices with seed phrase backups requires some education, but it eliminates your wallet provider as a single point of failure.
As with everything in bitcoin, there is no perfect solution, only different trade-offs that one can make. While there are plenty of examples of software and technology not quite working right and there are certainly challenges with securing physical items, in my experience, multisig with keys distributed is the most foolproof way to secure bitcoin, because nothing physical that you are protecting is a single point of failure—there are redundancies that allow you to make a mistake.
In the bitcoin-native economy, there is no reason any single company should be able to prevent you from accessing your bitcoin at any time. This is one of the promises of bitcoin, and these are the ideals we live by at Unchained Capital.
Collaborative custody goes one step further
In my time at Unchained, I’ve seen multisig save clients from losing hundreds of bitcoin, whether it be from forgetting a passphrase or not capitalizing it correctly, forgetting a device pin and/or having their device wipe the seed, losing seed phrases, firmware updates that wipe seeds off the devices entirely, and depositing bitcoin to old addresses where they lost a key.
Using a multisig wallet can provide some fault tolerance for your seed phrases, ensuring that if you lose one, it doesn’t immediately endanger your bitcoin. But collaborative custody takes this one step further, providing the benefits of multisig while reducing the number of keys and seed phrases you need to store, as well as providing self-custody support and a backup key should you ever need it. Get in touch with us to find out more, or set up a collaborative custody vault today.