May 9, 2023: Testing in Prod 💻🏗️
Eff it, we'll do it live
Can’t believe it’s already one year old. Hope everybody’s year was steddy.
Perhaps soon Do’s master plan will be unveiled? Today Terra’s stack of 1.8MM $CVX is on the move to destinations unknown.
DEVELOPING…
Yesterday we saw an interesting exchange on a website that was once free (as in beer).
It recalls our first read-through of the $crvUSD white paper, when we also couldn’t comprehend what the Curve math geniuses were thinking. But this turned out to be a reflection of our own position at the left end of the bell curve, so we simply kept quiet.
Curve’s reply recalled its long history of testing major launches in prod. It got us contemplating the entire concept.
For those who are not developers, “prod” refers to “production.” In a traditional web2 dev environment, it’s common to set up a few separate environments.
dev: Exclusively for development, where things can be broken without risk.
staging: Environment meant to closely mimic live conditions, used to test a final draft candidate before it is pushed live.
production: The final draft version used by the general userbase.
Blockchain development is slightly different. Several protocols simply deploy their entire suite to a testnet like Sepolia or Görli, then poke and prod it to see how it performs. These testnets provide a public method of deploying contracts, but without any value to their testnet ETH it’s a safe space to check performance without risking funds.
Curve mostly does not utilize testnets. For $crvUSD they were done solely for testing if Etherscan would verify contracts at the bleeding edge of Vyper development, but most Curve contracts have no footprint on testnets.
Instead, Curve prefers to test against a forked mainnet using ape framework, or historically, brownie. Node providers like Alchemy and Infura offer the ability to fork the tip of the live blockchain. This is great for previewing real transactions, as you can quickly simulate the exact state of the blockchain after you make a transaction. It’s like being able to always test against the `staging` branch without having to work to keep it up to date.
For complex deployments like Curve, a forked mainnet is clearly preferable. Curve has over 200 funded pools which users have deployed on mainnet, containing a similar number of complex and often highly customized ERC-20 tokens. Maintaining a similar architecture on a testnet would be a nightmare. Far simpler and more accurate to simply check things against what has actually been deployed.
Both testnet and mainnet-fork deployments have limitations though. It can be a lot of extra work to properly simulate important real-world conditions:
MEV
Arbitrage bots
Reentrancy attacks
Chain reorgs
Unforeseen price fluctuations
Targeted hacks
There are some solutions… you can write a script to reproduce historical data from, say, the depeg of $UST (the historical Terra event, not the upcoming US Treasuries event).
But what if you say eff it… we’ll do it live?
Human psychology is a funny thing. Post some contracts to Github, and nobody takes a look. Push them live and put funds into them, and it gathers attention. A bug bounty receives very few serious replies.
Put up a live contract and fund it with a million dollars of your own money, and suddenly you receive very serious attention from hackers who wouldn’t look twice at a bug bounty.
Ergo, we get to enjoy the spectacle of Mich dog fooding his own contracts, dropping what amounts to pocket change for him into contracts and seeing if anything goes wrong.
We find it hard to criticize the approach when he’s willing to personally risk what we’d consider a large sum of money.
So which is it, a highly productive testing strategy or a “mangled eff-up.”
Here’s our two cents…