Connect with us

Blockchain

Scaling connections with Ruby and MongoDB

Republished by Plato

Published

on

Coinbase

By Michael de Hoog

Coinbase was launched 8 years ago as a Ruby on Rails app using MongoDB as its primary data store. Today, the primary paved-road language at Coinbase is Golang, but we continue to run and maintain the original Rails monolith, deployed at large scale with data stored across many MongoDB clusters.

This blog post outlines some scaling issues connecting from a Rails app to MongoDB, and how a recent change to our database connection management solved some of these issues.

Global VM Lock

At Coinbase we run our Ruby applications using CRuby (aka Ruby MRI). CRuby uses a Global VM Lock (GVL) to synchronize threads so that only a single thread can execute at once. This means a single Ruby process can only ever use a single CPU core at once, whether it runs a single thread or 100 threads.

Coinbase.com runs on machines with a large number of CPU cores. To fully utilize these cores, we spin up many CRuby processes, using a load-balancing parent process that allocates work across these child processes. In the application layer, it’s hard to share database connections between these processes, so instead each process has its own MongoDB connection pool that is shared by that process’ threads. This means each machine has 10–20K of outgoing connections to our MongoDB clusters.

Blue-green deploys

Maintaining product velocity is essential at Coinbase. We deploy to production hundreds of times a day across our fleet. In the general case we use blue-green deployments, spinning up a new set of instances for each deploy, waiting for these instances to report healthy, before shutting down the instances from the previous deploy.

This blue-green deploy approach means we have 2x the count of server instances during these deploys. It also means 2x the count of connections to MongoDB.

Connection storms

The large count of connections from each instance, combined with the amount of instances being created during deploys, leads to our application opening tens of thousands of connections to each MongoDB cluster. Deploying during high traffic periods, when our application is auto-scaled up to handle incoming traffic, we would see spikes of almost 60K connections in a single minute, or 1K per second.

Hoping to reduce some of this connection load on the database, in March we modified our deployment topology, introducing a routing layer designed to transfer this load from the `mongod` core database process to a `mongos` shard router process. Unfortunately the connections were similarly affecting the `mongos` process and didn’t resolve the problem.

We experienced various failure modes from these connection counts, including an unfortunate interaction where the Ruby driver could cause a connection storm on an already degraded database (this has since been fixed). This was seen during a prolonged incident in April, as described in this Post Mortem, where we saw connection attempts above MongoDB’s 128K maximum to a single host.

MongoDB connection attempts to a single cluster, grouped by replica set member

Proxying connections

The vast amount of connections from our Rails application is the root problem; we had to focus on reducing these. Analyzing the total time spent querying MongoDB demonstrated these connections went mostly unused; the application could serve the same amount of traffic with 5% of the current connection count. The obvious solution was some form of external connection pooling, similar to PgBouncer for PostgreSQL. While there was prior art, there was no currently supported solution for connection pooling for MongoDB.

We decided to prototype our own MongoDB connection proxy, which we call `mongobetween`. The requirements were simple: small + fast, with minimal complexity and state management. We wanted to avoid having to introduce a new layer in Rails, and didn’t want to reimplement MongoDB’s wire protocol.

`mongobetween` is written in Golang, and is designed to run as a sidecar alongside any application having trouble managing its own MongoDB connection count. It multiplexes the connections from the application across a small connection pool managed by the Golang MongoDB driver. It manages a small amount of state: a MongoDB cursorID -> server map, which it stores in an in-memory LRU cache.

Results

Since rolling out the connection proxy, we’ve dramatically reduced the overall count of outgoing connections to MongoDB, by around 20x. Deploy connection spikes which used to hit 30K now hit 1.5K connections. The application steady state, which used to require 10K connections per MongoDB router, now only needs 200–300 connections total:

MongoDB connections drop significantly May 21st after deploying the proxy

Open source

Today we’re announcing that we are open-sourcing the MongoDB connection proxy at github.com/coinbase/mongobetween. We would love to hear from you if you are experiencing similar MongoDB connection storm issues and would like to chat about our solution. If you’re interested in working on challenging availability problems and building the future of the cryptoeconomy, come join us.

Source: https://blog.coinbase.com/scaling-connections-with-ruby-and-mongodb-99204dbf8857?source=rss—-c114225aeaf7—4

Blockchain

Coinbase Targets Direct Nasdaq Listing of its Class A Common Stock

Republished by Plato

Published

on

After the confidential submission of a draft registration statement to become a publicly-traded company announced last year, Coinbase has taken a major step to be listed on the giant US stock exchange – Nasdaq.

  • CryptoPotato reported earlier that Coinbase had submitted a confidential draft registration statement to the US Securities and Exchange Commission (SEC) to go public via a direct listing instead of an IPO. Initial estimations suggested that the potential valuation was at about $28 billion. 
  • Earlier today, the company announced that it had filed a registration statement on Form S-1 with the Commission “relating to a proposed public direct listing of its Class A common stock.” 
  • This signifies a vital step towards becoming a publicly-traded company. Coinbase plans to list its Class A common stock on the Nasdaq Global Select Market under the ticker symbol “COIN.” 
  • It’s worth noting, though, that while the registration statement has been filed, it has yet to become effective. As the company explained it:  
  • “These securities may not be sold, nor may offers to buy be accepted, prior to the time the registration statement becomes effective.” 

  • Furthermore, Coinbase asserted that its filing doesn’t “constitute an offer to sell or the solicitation of an offer to buy any securities, nor shall there be any sale of these securities in any state or jurisdiction in which such offer, solicitation, or sale would be unlawful prior to registration or qualification under the securities laws of any such state or jurisdiction.” 
  • The company reported that its net revenue results for 2020 were substantially larger than the 2019 numbers – $1.1 billion against $483 million. The expenses had also increased from $580 million in 2019 to nearly $870 million last year. 
SPECIAL OFFER (Sponsored)
Binance Futures 50 USDT FREE Voucher: Use this link to register & get 10% off fees and 50 USDT when trading 500 USDT (limited offer).

PrimeXBT Special Offer: Use this link to register & enter CRYPTOPOTATO35 code to get 35% free bonus on any deposit up to 1 BTC.

You Might Also Like:


Source: https://cryptopotato.com/coinbase-targets-direct-nasdaq-listing-of-its-class-a-common-stock/

Continue Reading

Blockchain

Polkadot, Ethereum Classic, IOST Price Analysis: 25 February

Republished by Plato

Published

on

Polkadot saw a bounce to $36 after touching $28 over the past few days, but it has once again sunk beneath the $34 mark. Ethereum Classic also found some resistance at the $12 mark, and IOST showed rising bearish pressure.

Polkadot [DOT]

Polkadot, Ethereum Classic, IOST Price Analysis: 25 February

Source: DOT/USDT on TradingView

The Supertrend indicator showed a sell signal for DOT after it slipped beneath the descending channel (cyan) and plunged to $28. However, since then it has made a recovery toward $34-but whether this is more of a bounce rather than a recovery is a pertinent question.

At the time of writing, trading volume did not really back the recent price rise, indicating that it was indeed a bounce. The Awesome Oscillator showed bearish momentum, but no real strength over the past few hours.

Key levels to watch out for are $34.5 and $36 above it. A rejection would indicate bearish strength, while a flip to support can be used to enter a long position on a retest.

Ethereum Classic [ETC]

Polkadot, Ethereum Classic, IOST Price Analysis: 25 February

Source: ETC/USDT on TradingView

Using the Fibonacci retracement tool for ETC’s drop from $18 to $9, some levels of importance are highlighted. The RSI showed bears were in control of the market over the past couple of days, as the RSI stayed beneath the neutral 50 value.

The Parabolic SAR climbed into overbought territory even as the price slipped beneath the 38.2% retracement level- which was not an encouraging sign for the bulls.

It is likely that ETC would continue to move lower, toward the $10.75 mark once more.

IOST

Polkadot, Ethereum Classic, IOST Price Analysis: 25 February

Source: IOST/USDT on TradingView

IOST was in a steady short-term decline. It attempted a recovery, on strong trading volume, to $0.059 from the depths of $0.039. However, bears were able to force the price lower once more.

An interesting aspect is a lack of buying strength when IOST began to slip after a rejection at $0.059. This showed that bears were in control. The levels that bulls would try to defend are the $0.044 and $0.039 levels.

A defense of either of these levels over the next couple of days would point toward weakening bearish pressure in the short-term, and another possible recovery back toward $0.059.


Sign Up For Our Newsletter


Source: https://ambcrypto.com/polkadot-ethereum-classic-iost-price-analysis-25-february

Continue Reading

Blockchain

Inside the blockchain developer’s mind: Koinos approaches testnet

Republished by Plato

Published

on

Cointelegraph is following the development of an entirely new blockchain from inception to mainnet and beyond through its series, Inside the Blockchain Developer’s Mind. In Part Four, Andrew Levine of Koinos Group discusses some of the challenges the team has faced since identifying the key issues they intend to solve.

Earlier in this series I outlined three of the “crises” that are holding back blockchain adoption; upgradeability, scalability, and governance.

In this post I will summarize the solutions we’ve developed to these problems, which we will be showcasing in the upcoming Koinos testnet planned for the second quarter of 2021.

Since that series Koinos Group has successfully launched a token, KOIN, as a proof of work mineable token on Ethereum. By using proof of work to distribute the initial token supply we were able make the token accessible to early adopters and forgo an ICO.

Assessing the ICO model

ICOs and similar token sale tools, while not without their use cases, have created their own crisis within the space by misaligning incentives before development even begins. The issue is not with the ICO as a tool, but what happens when a team is financially rewarded before they have even shipped a product.

While so many projects have followed in the footsteps of Bitcoin, it’s surprising how few have replicated arguably the most successful aspect of its launch; a token distribution exclusively through proof of work.

The benefit of this approach is that it ensures with algorithmic certainty that the people behind the blockchain have no advantage in acquiring the token. In short, everyone, no matter who they are, has to make a financial sacrifice in order to acquire that token and the scale of that sacrifice is determined by some neutral third party. In the case of proof of work, that neutral third party is the manufacturer of hardware.

For Koinos Group, that means we had to spend money to acquire our token just like everyone else. In fact, because we have to spend most of our time developing the product, we are even at a disadvantage relative to professional miners. So we have to keep working to add value to the protocol if we’d like to get a return on our investment.

Proof of work algorithms are not without their problems, but we mitigated those in a few ways.

  • First, the mainnet will be governed by a totally different consensus algorithm that won’t be proof of work or proof of stake, so any attempt to develop an ASIC would be a waste of resources.
  • Second, we made the algorithm GPU resistant.
  • Third, we released this token long before releasing our mainnet. In fact, we released the token long before we had even completed development of our framework. Without a functional product, this token becomes a way for people who believe in our team and who share our vision for a fee-less smart contract platform to acquire the token at a reasonable cost.

Rapid rate of improvement

Part of what makes this launch strategy work is the innovative property set of Koinos. We built Koinos totally from scratch, not around any single feature like transactions per second or sharding, but with the goal of creating a blockchain that would improve at a much more rapid rate than any other blockchain out there.

In our experience developing the Steem blockchain, the need to execute hard forks was the single biggest factor holding back progress. If we wanted to eliminate that bottleneck, we reasoned, moving as much of the system code as possible into smart contracts that could be upgraded in-band would do the trick.

That’s why the Koinos blockchain framework contains only the most basic blockchain features (called “thunks”) like contract input/input, getting parameters, and writing to the database. All of the more complex features that people are more familiar with (consensus algorithm, accounts, resource management, governance, etc.) have been moved into modular WASM smart contracts running in the virtual machine that can be upgraded without a hard fork.

Because all behaviors are now coded in distinct “modules” that can be individually “upgraded” we call this feature modular upgradeability.

As a result of modular upgradeability, any behavior can be added to the blockchain without a hard fork because individual upgrades can be distributed in blocks and transactions that are pushed to the network much like an operating system patch, but with the added benefit of an on-chain record of the entire upgrade path.

By moving nearly all of the system code of the blockchain to smart contract modules that can be upgraded without a hard fork we have made Koinos into a blockchain that derives its strength not from the features it is born with, but based on its ability to rapidly acquire new and better features faster than anything else out there.

This is why we call Koinos the first blockchain capable of evolution.

Microservices

Modular upgradeability was just the first major technical innovation that we developed to make Koinos less monolithic and an order of magnitude more upgradeable. Just like there is code that does not need to be implemented natively (in the blockchain itself) but that can be implemented as smart contracts (most of it in fact), there is plenty of code that does not need to be implemented either natively or as smart contracts and can instead be implemented as microservices.

Microservice architectures have many benefits which is why this has become the industry standard for modern software development, but one major benefit is scalability because individual services can be scaled up without having to scale up the entire system. This can dramatically reduce the cost of running a network while improving both the speed and quality of improvements to that network. As a result of historical accidents, blockchain stacks appear to be the last to adopt this new standard as Koinos will be the first blockchain built on a microservice architecture.

This creates amazing new opportunities for developers who will be able to build application specific microservices for Koinos that will help them run their nodes, and their applications, more efficiently; and as a consequence deliver better user experiences. Best of all, this will make Koinos node operation more accessible, thereby improving decentralization, and enabling the network as a whole to run more efficiently so that developers and their end-users can get more out of their decentralized applications.

Multi-language support

Another benefit of a microservice architecture is that individual microservices (basically small programs) can be written in the best (fastest, most secure, best libraries, etc.) programming language for the job, a capability we also wanted to offer for smart contract developers. But in order to take advantage of this trait we needed to develop a way for these small programs written in different languages to “talk” to one another in a way that conformed to the unique needs of a decentralized network. To solve this problem we created a cross-language serialization framework named Koinos Types.

Koinos Types is like the Rosetta Stone for blockchain data structures. It allows programs written in different languages to talk to one another in a simple and unified way by giving them access to the same objects (the “building blocks” of modern programming languages). Koinos Types allows for the interpretation of Koinos (i.e. blockchain) data structures in practically any programming language which will be extremely useful for the development of blockchain-related microservices, clients, and smart contracts.

Koinos Types solves a number of problems. It helps us add multi-language support to Koinos more generally (including for smart contracts), it enables microservices to communicate with one another, and it makes it far easier to develop and update client-libraries. While modular upgradeability and the microservices architecture alone make Koinos far more upgradeable than any other blockchain, Koinos Types takes that upgradeability to another level. That’s why we were so excited to make Koinos Types the first piece of Koinos that we open sourced.

As you can see, ensuring that Koinos can improve at a more rapid rate than any other blockchain isn’t about any one feature.

  • It’s about getting the incentives right from the beginning.
  • It’s about ensuring that the blockchain has modular upgradeability.
  • It’s about modularizing the very architecture itself as microservices.
  • And it’s about making sure that developers operating at every level of the stack (not just smart contracts) are able to use the programming languages they already know and love.

The views and opinions expressed here are solely those of the author and do not necessarily reflect the views of Cointelegraph.com. Every investment and trading move involves risk, you should conduct your own research when making a decision.

Andrew Levine is the CEO of Koinos Group, where he and the former development team behind the Steem blockchain build blockchain-based solutions that empower people to take ownership and control over their digital selves. Their foundational product is Koinos, a high-performance blockchain built on an entirely new framework architected to give developers the features they need in order to deliver the user experiences necessary to spread blockchain adoption to the masses.

Source: https://cointelegraph.com/news/inside-the-blockchain-developer-s-mind-koinos-approaches-testnet

Continue Reading
Blockchain3 days ago

Ankr adds Eth2 futures (fETH) to its staking system

Blockchain4 days ago

Ripple now registered as a Wyoming business

Blockchain5 days ago

Elon Musk Explains to Peter Schiff What Money Is

Blockchain4 days ago

Former BoE, BoC Governor Mark Carney joins Stripe board of directors

Blockchain4 days ago

Litecoin, Cosmos, Tezos Price Analysis: 21 February

Blockchain3 days ago

Peter Schiff Now Discusses Bitcoin More Often Than His Beloved Gold

Blockchain4 days ago

A Review of BTCGOSU — Reviewer of Crypto Casinos

Blockchain3 days ago

Kraken Daily Market Report for February 21 2021

Blockchain3 days ago

DeFi Protocol Primitive Finance Self Hacks to Prevent Exploit

Blockchain3 days ago

Long Blockchain Corp has officially been delisted by SEC

Blockchain4 days ago

The Many Theories Of Elon Musk Being Satoshi Nakamoto

Blockchain3 days ago

NFT Platform Ethernity to Launch IDO on Polkastarter

Blockchain4 days ago

Is Ethereum heading to another ATH?

Blockchain5 days ago

3 key factors that propelled Ethereum to $2,000 for the first time ever

Blockchain4 days ago

Banks will be required to work with crypto, e-money and CBDCs to survive

Blockchain2 days ago

Bitcoin falls to $45K in sequel to 20% BTC price crash

Blockchain5 days ago

Kraken Daily Market Report for February 20 2021

Blockchain3 days ago

MoneyGram suspends Ripple partnership, citing SEC lawsuit

Blockchain4 days ago

Today 11:40 am EST: First Bitcoin Elite NFT Art Drop

Blockchain3 days ago

Kraken users demand refunds over flash-crash liquidations

Trending