uvloop makes asyncio fast. In fact, it is at least 2x faster than nodejs, gevent, as well as any other Python asynchronous framework. The performance of uvloop-based asyncio is close to that of Go programs.
At Paxos, we make heavy use of asyncio because it’s more performant and also because it’s a better fit for our architecture. Our product synchronizes transaction data from many different clients and is inherently asynchronous because it’s a distributed system.
asyncio is great for maximizing throughput, but we also care about reducing latency. For latency, your database can be a major bottleneck. When using best practices like a proper index and SSDs, a ~10 ms response time for a database query is considered good.
RAM is orders of magnitude faster, but simple key-value stores don’t allow for many of the features software engineers rely on. Multiple indices and basic type checking are very useful. ORDER BY, LIMIT, and OFFSET, are nice to have as well.
We’ve been impressed with redis due to its performance, advanced features and documentation. So, we wrote a redis-backed object mapper called subconscious that relies on asyncio and redis’ powerful primitives. You get the benefits of a database, with the performance of RAM! It’s like a bare-bones version of SQLAlchemy that will never support JOIN operations.
You can define a model like this (for more details, check out the official repo):
MALE = 'male'
FEMALE = 'female'
uuid = Column(primary_key=True)
name = Column(required=True)
age = Column(index=True, type=int, sort=True, required=True)
gender = Column(index=True, enum=Gender)
country_code = Column(index=True)
Then you can query and use your model like this:
some_user = await User.load(db, some_uuid)
if some_user.age > 65:
# apply senior citizen discount
price -= 10.00
You’ll also find lots of features you’ve come to expect from traditional ORMs. Here’s how you’d send a promotion to your US, college-age, male users:
async for user in User.filter_by(
age=[18, 19, 20, 21, 22],
In order to demonstrate how powerful and easy to use this is in practice, we’ve thrown together a pastebin-like website in ~100 lines of code:
We load-tested this app on Microsoft Azure’s cheapest instance (called an F1, which has 1 core and 2 GB RAM), and the numbers are great:
- 10k pageviews took ~41s
- Median pageview was ~19ms
- 90% of pageviews return in < 22ms
Redis can be saved to disk, but has much weaker persistence vs a traditional database. Transactions and rollbacks are complicated, and there are less guarantees when it comes to data integrity. Some common use-cases:
- Ephemeral data that can be replayed from a log (so persistence isn’t a big concern)
- Frequently-updated data: writing to RAM is *much* faster than writing to disk
- When latency is extremely important
- When the same data is queried regularly, the volume of data is low, and/or you have a large hardware budget
For low-throughput applications or applications that lend themselves to horizontal scaling, these features may not provide enough benefit to get excited about. But, for high-throughput apps where latency matters, the performance increase can be substantial.
Subconscious is open-sourced using an MIT license, so feel free to use it for your own projects!
Interested in working with these technologies (and more)? We’re hiring software engineers.
I spoke at a recent New York Python Meetup Group event hosted by Stack Overflow that focused on a library we open-sourced called dsert. At Paxos, we use it for API testing, and you might want to as well. Using dsert we can test all the fields of a python dictionary, even when we don't know all of their values. This ensures that tests are as explicit as possible. You can watch the full video below, or on our Paxos YouTube page.
Many people across the technology and financial industries are dubbing blockchains as the greatest innovation since the Internet. However, a blockchain is comprised of a bunch of technologies that are actually pretty old. The biggest surprise when it comes to Bitcoin may be that it wasn’t invented a decade earlier using dial-up internet!
Most blockchains use the six major technologies below. In this post, we will examine each technology and explain the role they play in a blockchain.
The magic of asymmetric encryption is that signatures accomplish the following:
- Prove that the signer had access to the private key
- Do not reveal the private key
- Are trivial to verify, yet impossible to forge/alter
Bitcoin uses the secp256k1 parameters of the Elliptical Curve Digital Signing Algorithm (ECDSA). ECDSA was invented in 1985 and became an ISO, ANSI, IEEE and FIPS standard in 1998-2000. The major advantage ECDSA has over RSA is that ECDSA uses much smaller keys and signatures to achieve the same level of security. In other words, Bitcoin would have been possible with RSA... which was invented in 1977!
One of the main use cases for asymmetric encryption is public-key encryption. For example, Alice can encrypt a message using Bob’s known public key and send it to Bob over an untrusted network so that Bob can then decrypt the message using his private key. This feature of public/private key cryptosystems is not really used in Bitcoin.
Hash functions take input data of arbitrary size and deterministically map it to an output of fixed size (typically smaller than the input size) that resembles random data. A key property of a hash function is collision resistance.
You may have noticed that each transaction and block in Bitcoin is represented by 64 hexadecimal characters. That’s because these “IDs” are calculated deterministically by serializing the transaction/block contents into bytes and then hashing those bytes (twice) using SHA-256. The result is the transaction/block hash.
This provides a convenient integrity check on transactions/blocks. Just as an asymmetric signature cannot be altered by a dishonest actor, the contents of a transaction/block cannot be tampered with due to collision resistance. This provides a very useful guarantee to all participants that their version of history is the same as all other participants on the blockchain. If two sources share the same current block hash, then they know they share every single input/output in every previous transaction/block.
Another use of hash functions in Bitcoin is that public keys are hashed in order to determine a Bitcoin address. This is a defensive protection against the future invention of a quantum computer that could break ECDSA (learn more here).
Bitcoin transaction/block hashes and merkle trees use two rounds of SHA-256, while address derivation uses two rounds of SHA-256 and one round of RIPEMD-160. SHA-256 was first published in 2001 and RIPEMD-160 was first published in 1994. Hash functions have been around a lot longer.
In order to package transactions into blocks, Bitcoin uses a Merkle Tree. This data structure takes a list of transactions and combines them using a binary tree structure, where the root node is called the Merkle Root. The killer feature of the Merkle Tree is that proof a leaf was included in the Merkle Root takes O(log(n)) space. Using this technique makes it possible to run a very secure Simple Payment Verification (SPV) bitcoin wallet on your phone without storing 100+ GBs of blockchain data.
The Merkle Tree was patented in 1979 and is used in distributed file systems like IPFS, file-sharing systems like BitTorrent and NoSQL databases like Cassandra.
In order to prevent double-spends (a key feature of a blockchain), you have to be able to quickly perform the following two database operations:
- Lookup if a transaction has already been spent
- Insert a new valid transaction
While NoSQL databases have taken off in popularity in recent years (due primarily to their ability to handle extreme scale), it is important to remember that NoSQL databases actually pre-date relational databases.
P2P Communication Protocol
Having nodes communicate directly with one another (as opposed to using a trusted third-party) is unlike most applications we use on a daily basis. However, it isn’t new.
Napster’s 1999 release is probably the most commonly known P2P network, but USENET predates it by two decades.
The internet provides an excellent network for P2P protocols. Gossip Network Protocols have been used in many NoSQL Databases including Amazon Dynamo, Cassandra and Riak.
Proof of Work
Proof of work (PoW) is a clever application of hash functions. It works by calculating the hash of a message, along with many different nonces, until you find a resulting digest that meets a rare criteria. Since each hash is equally unlikely to meet that criteria, specifying a hard-to-meet criteria (perhaps a hash that starts with several leading 0s) is a way to prove that someone spent their CPU cycles. Also, a correct solution will be trivial to verify.
By using proof of work to achieve consensus, it becomes unfeasibly expensive to attack the Bitcoin blockchain. This is a cornerstone feature for a permissionless network built on anonymity and strong distrust of other participants.
Some people find PoW to be inefficient, since the Bitcoin network spends enormous computing resources doing work that has no other value.
PoW was first invented in 1993, but was made famous in 1997 by Adam Back’s Hashcash.
Evolution of Blockchain Technology
The fact that blockchains use old technologies should not be taken to mean that they represent solved problems or are easy to deploy. After all, it isn’t the technologies in a blockchain that make it useful, it’s the clever way in which they interact. Scale, privacy, security and fault tolerance are all hard problems in computer science. To make things more complicated, blockchains are typically used to handle mission-critical operations (such as settling financial transactions like we are doing at Paxos), where serious bugs may not be acceptable to end-users.
A modern blockchain must optimize tradeoffs while making as few sacrifices as possible, a difficult and delicate balancing act.
We are looking for talented engineers to help us revolutionize how assets move through the global financial system. Apply now to join our growing team in NYC!
Part one of this two-part series took an in-depth look at the Bitcoin blockchain and the properties that add value in a variety of digital transactions. In part two, we look at the key properties of a blockchain, how they apply to public and private blockchains and the value of private blockchains for trading assets beyond bitcoins.
It’s always easiest to think of a blockchain as a subset of a database. To begin, let’s take a look at the six core properties of a blockchain which allow it to operate without a central authority or third-party intermediary:
1. Provably Consistent: No conflicting transactions within the database and there are rules as to what can go in the database. Those rules are respected and the consistency of the database can be checked and proven.
2. Append-only: Database can only have new data added, old data cannot change or be deleted.
3. Ownable: Certain data in the database can be owned and only the owner may operate on that data through use of proofs.
4. Highly Available: Data must be available to whoever needs to check its consistency with database rules.
5. Canonical: There is exactly one true database. If there are multiple versions, it’s easy to determine which one is actually the true one.
6. Practically Immutable: It costs an impractical amount of time and/or money to subvert any of the above properties.
Bitcoin: The First Database with No Central Authority
There have been many databases with at least some of these properties. However, no database combined all of these properties until the advent of Bitcoin in 2009, making it the first distributed database that did not need a central authority to operate. As we mentioned in part one, having no central authority is desirable for many reasons, including faster transaction speeds, increased privacy and reduced transaction friction.
Let’s look at how Bitcoin establishes each property:
Bitcoin has rules for its database called libconsensus and they are literally codified in C++ (note: it took Bitcoin developers several years to do this). This provides provable consistency as any node on the network can check that the entire blockchain database respects the rules.
The Bitcoin blockchain is a database where each block contains the fingerprint of a previous block. Thus, the database can only append entire blocks at a time. Further, each block’s digital fingerprint changes if anything in the block changes. This makes it very difficult to alter a previous block.
Most coins (and thus, data) in the Bitcoin blockchain require proof of a private key. That means spending from a previously unspent output (aka UTXO) requires some proof that you have permission to spend it. This is done through public key cryptography and more specifically through something called the Elliptical Curve Digital Signature Algorithm (ECDSA).
Since Bitcoin is a decentralized public network, anyone can connect to the network and download the entire blockchain (~100GB) to check for consistency. As of this writing, there are approximately 5,500 nodes from which the blockchain can be downloaded and checked.
Bitcoin solves the double spending problem through the use of something called proof-of-work (POW). The main property of proof-of-work is that it requires a tremendous amount of computation to create, yet very little computation to validate. Each block in the Bitcoin blockchain requires proof-of-work and all the computing power in the world dedicated to doing this takes roughly 10 minutes to find. This means that to create an alternate version of the blockchain would cost the same amount of computing power.
The main way Bitcoin provides immutability is through proof-of-work. At least 51% of the global Bitcoin network’s computing power is needed in order to be able to subvert Bitcoin’s canonical or append-only properties.
Similarly, you would need to bring down all 5,500 nodes (some of which are very hard to locate) on the network today to subvert the high availability of the Bitcoin database. The code for consistency is published and widely available and to subvert that part, each of the 5,500 nodes would have to be changed in the same way. Again, this is an unfeasibly expensive prospect. Additionally, public key cryptography is known to be an extraordinarily difficult problem to solve and the only known way to subvert it would be with computational power greater than all computing power in the world for billions of years.
It’s important to remember that the absence of any of these properties would necessitate a central authority. If you can have conflicting transactions, transfer of value would be fraught with risk. If database history can be changed, again, transfer of value would be fraught with risk. If there was no ownership, there would be no transfer of value. If there was no public availability of the database, then no one would be able to trust the database or transact on it. Without canonicity, double-spending would be possible causing transfers of value to be fraught with risk. Finally making everything prohibitively expensive to alter gives the Bitcoin network strong security.
Applying Blockchain Properties to a Private Blockchain
In contrast to a public blockchain like Bitcoin, all participants are known in a private blockchain. There are onboarding processes for each node and the absence of anonymity adds security, trust and data integrity to the shared database. These characteristics make private blockchains the preferred option for applications within the financial services industry.
Like public blockchains, private blockchains can also operate without a central authority. Let’s take a closer look at how private blockchains can satisfy each property:
Provably Consistent & Highly Available
Generally, databases are said to be provably consistent if they can be constantly audited. Distributing the database to each node in a private network would certainly make this possible. The only requirements would be distributing a set of rules that dictate what data can be added and that the data indeed conforms to those rules. Making the data available is a problem that’s been largely solved. Most websites, for example, host the same data in different data centers around the world to make their websites available even when a couple hosts fail. Similar technologies exist for databases, even without the gossip network and large number of nodes that bitcoin uses.
Append-only & Canonical
Making a database append-only requires some checkpoints, which Bitcoin does through blocks. Creating an append-only database is something that already exists with a class of database storage units called WORM (Write once, read many). That, of course, relies on hardware properties to ensure that nothing is altered afterwards. In a private networking context, we will need a tool that is very expensive to reverse-engineer or change. This can be done a number of ways, including a private proof-of-work algorithm, merge-mining with Bitcoin or even signatures of parties that would otherwise not cooperate. The act of figuring out what’s canonical is what we would call consensus modeling. At Paxos, as befitting our name, this is an area of research where we figure out the tradeoffs to various consensus models.
Ownable & Practically Immutable
Ownability and practical immutability can be achieved using various forms of public key cryptography. Counterfeiting is virtually impossible (very expensive) and gives us practical immutability if the known parties of a private network sign.
Value of Private Blockchains for Trading Assets
For assets that are traded and settled digitally on a private blockchain (stock, bonds, etc.), the absence of a central authority means no third-party intervention is required. There are certainly regulatory considerations to retrofit into our new model. However, absent regulatory concerns, we can trade and settle traditional assets on a private blockchain with a similar level of security, efficiency and cost-savings as moving bitcoins on a public blockchain.
For assets that are either physical like gold or in another system like US dollars, third-parties are required to act as depositories. However, unlike a central authority, they are merely guarantors of the underlying asset on the private blockchain should the asset be requested. As a result, such third-parties don’t have to take on counterparty risks, operating costs or even privacy liabilities associated with being a central authority.
Blockchains cut out risks and costs present when a central authority is part of the trade settlement process. Private blockchains can enable faster settlement speeds, reduce liabilities and counterparty risk and bolster security, helping entire financial systems operate more efficiently.
We are looking for talented engineers to help us revolutionize how assets move through the global financial system. Apply now to join our growing team in NYC!
I spoke at a recent New York Python Meetup Group event hosted by Stack Overflow that focused on Async Programming. My talk was about thirty minutes long and you can watch the full video below. The video is also available on our Paxos YouTube page.
“Blockchain” has been a popular term in the financial industry over the last four years. Countless articles have been written about its benefits, how this innovative technology can change the way business is done and what it means for the future.
Given the abundance of information and commentary out there, how do you separate hype from substance? What’s the actual end benefit of blockchain technology and does it have any value beyond Bitcoin?
This two-part series cuts through the noise, deconstructing what a blockchain is at its most fundamental level and exploring the technology’s uses in the financial services industry. Part one examines the Bitcoin blockchain and identifies inherent blockchain properties that add value for data integrity, privacy and security. Part two (due out in February) will dig deeper into the benefits of blockchain technology independent of Bitcoin and what a blockchain-like database with systemic integrity can bring to a business.
Blockchain Anatomy 101
Blockchain is often introduced as the technology behind Bitcoin. Though true, this simplified explanation obscures the fact that blockchain technology is a rigorous system that creates certain desirable properties and has a variety of potential applications beyond Bitcoin.
At the most fundamental level, it’s easiest to think of a blockchain (also referred to as a distributed ledger) as a subset of a database, or a data store with special characteristics. In the case of Bitcoin, the basic unit used in the blockchain are transfers of value called transactions. Transactions in this context are similar to the common English usage of the term, that is, an asset (in this case bitcoins) being transferred from one owner to another.
These transactions almost always require signatures which add easy-to-prove authorization. This is done through the use of public-key cryptography, and in Bitcoin’s case ECDSA (Elliptical Curve Digital Signature Algorithm). Such signatures are near-impossible to forge as they require immense amounts of computation (think every computer in existence for millions of years) to create without the private key.
Transactions are grouped into what are called blocks. Blocks are the mechanism by which the data store can be checked for consistency. This is done through two mechanisms.
First, all blocks point to a previous block. This allows them to be well-ordered and is the inspiration for the term blockchain. Each block extends a chain of blocks with the obvious exception being the first block, which is called the genesis block. Having an ordering of blocks allows for an ordering of transactions. This is important for auditing and validation purposes.
Standard composition of a blockchain, from newest block to the genesis block
Second, each block must not only be self-consistent (no conflicting transactions within the block) but also consistent with the network rules (no conflicting transactions with transactions in previous blocks in the chain). This allows anyone with access to the entire blockchain to check that the rules of the network are respected. In addition, any capable machine can check that each new transaction (and thus, each new block) is consistent with all the transactions that came before. In short, any machine on the network has the ability to check the integrity of the database with respect to the network rules.
Bitcoin: The First Decentralized Digital Payment System
The blockchain, combined with something called proof-of-work, gives Bitcoin an extremely desirable property. Because the complete record of every transaction is shared publicly, decentralized and well-ordered through blocks, nobody can cheat by spending bitcoins they don’t have (double spending) or creating bitcoins that don’t exist. There’s a whole network that enforces which blocks/transactions are valid without a central authority. Essentially, Bitcoin is a digital payment system that eliminates the need to trust a third-party intermediary to execute transactions.
Think about that for a moment. Most digital payment systems require you to trust third-parties like banks and credit card providers. Conversely, in-person methods of payment like cash, gold or barter do not require trusted third-parties but they are non-digital and cumbersome in nature. Bitcoin has combined the best characteristics from both worlds.
Eliminating third-party intermediaries from the transaction process, as Bitcoin does, is valuable for three key reasons. First, third-parties have to make money and traditionally, they do this by adding cost to the system. For example, credit card providers take up to 3% of the transaction as fees for their service. Second, using a third-party intermediary adds risk. This may not be apparent when using reputable companies like Visa or Paypal, but the payment space is littered with companies that have gone bankrupt, had accounts seized or hidden insolvency by cooking the books (see Flooz, E-gold or Pay by Touch). Finally, the involvement of third-parties reduces privacy. An In-person cash transaction is nobody’s business but the two people involved in the exchange. When a third-party enters the picture, they gain access to sensitive personal and financial information.
In addition to its strengths as a decentralized digital payment system, the Bitcoin blockchain is also valuable in that due to proof-of-work, the ledger is immutable (for all practical purposes). That means no one can go back and change numbers around to make things look different. This is great for auditing and ensuring systemic integrity.
Blockchain and the Global Financial System of the Future
It’s important to remember that third party intermediaries aren’t present only in consumer transactions like buying groceries with a credit card. Thousands of times every second, third-parties help execute the buying/selling of stocks, derivatives, currency, precious metals and other assets. In short, most transactions today require both a third-party intermediary and centralized infrastructure that adds cost, risk and privacy-loss to the transaction.
The question, then, isn’t about the utility of blockchains in financial transactions, but whether it is possible to use blockchain's properties to increase privacy while decreasing cost and risk when executing these transactions. When discussing the financial applications of blockchains, there is less interest in the technical specifics as they pertain to Bitcoin. Rather, the excitement is in how blockchain technology’s inherent desirable properties can be applied to modernize how assets (beyond bitcoin) move through the global financial system.
Part two, set for release in February, digs deeper into the benefits of blockchain technology (digital exchange of assets without a central authority, practical immutability) without the mechanisms of Bitcoin and what a blockchain-like database with systemic integrity can bring to a business.
At Paxos, we are using blockchain technology to build the next-generation infrastructure that will power capital markets for years to come. Our blockchain platform, Bankchain, streamlines and automates post-trade settlement, the process that underpins and serves as the foundation for the global financial system.
Although trading speeds, regulations and the complexity of assets have changed, settlement has not. Today’s post-trade settlement systems were designed decades ago, many during the Nixon administration. Disrupting deeply entrenched global financial operations and processes is a huge challenge. We need the best engineers to solve the complex problems facing today’s capital markets and help us to modernize how assets move through the global financial system.
Our engineers come from a variety of backgrounds ranging from top financial institutions to innovative startups. Many of our engineers have also worked on innovative projects within the Bitcoin and blockchain community before joining Paxos. Regardless of where they come from, we strive to hire the best and expect our engineers to maintain the highest standards.
Paxos Engineering Core Values
Our core values guide the engineering team’s work and drive our culture of pragmatic innovation.
Paxos engineers don’t wait around to be assigned tasks, they take the initiative. We would rather have our engineers ask for forgiveness than permission.
Our engineers see their projects and responsibilities through to the end and are always accountable to one another.
Don’t Be Afraid of Failure
Paxos engineers are never afraid to experiment with new ideas. We don't consider churn a bad thing and our engineers are encouraged to refactor code and make it better. We expect that many of our ideas won’t work and that the ones that do will need a lot of iterations. We use every instance as an opportunity for improvement.
Be a Force Multiplier
"10X engineer" is a common term at tech companies, but isn't in our vocabulary at Paxos. We expect our engineers to be Force Multipliers. We do the grunt work and automate processes to make each other more productive and effective.
Keep It Simple
We believe in writing simple readable code and using the right tool (framework, language) for the job. We only choose technology because it fits a business need.
Committed to Excellence
Our engineers do not accept “good enough” and understand that there are no shortcuts to success. We have high standards and do everything in our power to ensure that our products, behavior and thinking meet those standards every day.
Good ideas can come from anywhere. At Paxos, no engineering idea goes unheard. We treat everyone equally and with the utmost respect. We believe that the best teams allow everyone equal time to convey their opinions and listen actively to those expressed by others.
Since we began our journey of discovery with customers over a year ago, we have fielded a wide variety of questions about blockchain and distributed ledger technology (DLT). It has become increasingly evident to us that while there is a lot of excitement about blockchain technology, there is also a lack of clarity on its applications and potential for disruption.
As engineers and blockchain evangelists committed to openness, we look forward to educating the broader engineering community about this innovative technology. We are big proponents of open source and passionate about sharing our code, perspectives, challenges and learnings. The Paxos Engineering Blog is the first step in sharing our knowledge with the community.