Connect with us

Blockchain

Scaling connections with Ruby and MongoDB

Published

on

Coinbase

By Michael de Hoog

Coinbase was launched 8 years ago as a Ruby on Rails app using MongoDB as its primary data store. Today, the primary paved-road language at Coinbase is Golang, but we continue to run and maintain the original Rails monolith, deployed at large scale with data stored across many MongoDB clusters.

This blog post outlines some scaling issues connecting from a Rails app to MongoDB, and how a recent change to our database connection management solved some of these issues.

Global VM Lock

At Coinbase we run our Ruby applications using CRuby (aka Ruby MRI). CRuby uses a Global VM Lock (GVL) to synchronize threads so that only a single thread can execute at once. This means a single Ruby process can only ever use a single CPU core at once, whether it runs a single thread or 100 threads.

Coinbase.com runs on machines with a large number of CPU cores. To fully utilize these cores, we spin up many CRuby processes, using a load-balancing parent process that allocates work across these child processes. In the application layer, it’s hard to share database connections between these processes, so instead each process has its own MongoDB connection pool that is shared by that process’ threads. This means each machine has 10–20K of outgoing connections to our MongoDB clusters.

Blue-green deploys

Maintaining product velocity is essential at Coinbase. We deploy to production hundreds of times a day across our fleet. In the general case we use blue-green deployments, spinning up a new set of instances for each deploy, waiting for these instances to report healthy, before shutting down the instances from the previous deploy.

This blue-green deploy approach means we have 2x the count of server instances during these deploys. It also means 2x the count of connections to MongoDB.

Connection storms

The large count of connections from each instance, combined with the amount of instances being created during deploys, leads to our application opening tens of thousands of connections to each MongoDB cluster. Deploying during high traffic periods, when our application is auto-scaled up to handle incoming traffic, we would see spikes of almost 60K connections in a single minute, or 1K per second.

Hoping to reduce some of this connection load on the database, in March we modified our deployment topology, introducing a routing layer designed to transfer this load from the `mongod` core database process to a `mongos` shard router process. Unfortunately the connections were similarly affecting the `mongos` process and didn’t resolve the problem.

We experienced various failure modes from these connection counts, including an unfortunate interaction where the Ruby driver could cause a connection storm on an already degraded database (this has since been fixed). This was seen during a prolonged incident in April, as described in this Post Mortem, where we saw connection attempts above MongoDB’s 128K maximum to a single host.

MongoDB connection attempts to a single cluster, grouped by replica set member

Proxying connections

The vast amount of connections from our Rails application is the root problem; we had to focus on reducing these. Analyzing the total time spent querying MongoDB demonstrated these connections went mostly unused; the application could serve the same amount of traffic with 5% of the current connection count. The obvious solution was some form of external connection pooling, similar to PgBouncer for PostgreSQL. While there was prior art, there was no currently supported solution for connection pooling for MongoDB.

We decided to prototype our own MongoDB connection proxy, which we call `mongobetween`. The requirements were simple: small + fast, with minimal complexity and state management. We wanted to avoid having to introduce a new layer in Rails, and didn’t want to reimplement MongoDB’s wire protocol.

`mongobetween` is written in Golang, and is designed to run as a sidecar alongside any application having trouble managing its own MongoDB connection count. It multiplexes the connections from the application across a small connection pool managed by the Golang MongoDB driver. It manages a small amount of state: a MongoDB cursorID -> server map, which it stores in an in-memory LRU cache.

Results

Since rolling out the connection proxy, we’ve dramatically reduced the overall count of outgoing connections to MongoDB, by around 20x. Deploy connection spikes which used to hit 30K now hit 1.5K connections. The application steady state, which used to require 10K connections per MongoDB router, now only needs 200–300 connections total:

MongoDB connections drop significantly May 21st after deploying the proxy

Open source

Today we’re announcing that we are open-sourcing the MongoDB connection proxy at github.com/coinbase/mongobetween. We would love to hear from you if you are experiencing similar MongoDB connection storm issues and would like to chat about our solution. If you’re interested in working on challenging availability problems and building the future of the cryptoeconomy, come join us.

Source: https://blog.coinbase.com/scaling-connections-with-ruby-and-mongodb-99204dbf8857?source=rss—-c114225aeaf7—4

Blockchain

U.S. Treasury Targets Stablecoins in Latest Regulatory Risk Assessment

Published

on

As regulatory pressure mounts in the U.S., policymakers are putting stablecoins at the top of their agendas.

Citing “people familiar with the matter,” Bloomberg has reported that officials are crafting a policy framework set to be released in the coming weeks. Their primary concern is ensuring that investors can reliably move money in and out of tokens, it added.

The anonymous insiders are worried that a “fire-sale run on crypto assets could threaten financial stability and that certain stablecoins could scale up dangerously fast.”

Strengthening Regulatory Efforts

The Financial Stability Oversight Council is also preparing a formal review into whether stablecoins pose an economic threat.

The officials are focusing on how stablecoin transactions are processed and settled and whether market conditions have an impact, it added. Tomicah Tillemann, global head of policy at a crypto fund run by venture capital giant Andreessen Horowitz, commented:


ADVERTISEMENT

“It is significant and very consequential that we are witnessing early steps to create a regulatory framework around digital assets. That’s a big deal.”

The report, when released, will go to the President’s Working Group on Financial Markets. The body includes key agency heads such as Treasury Secretary Janet Yellen, Federal Reserve Chair Jerome Powell, and Securities and Exchange Commissioner Chair Gary Gensler.

In late July, Yellen called for urgency in regulating stablecoins after stating that they are not adequately supervised. Gary Gensler echoed the sentiment in early August, stating that regulators must act to protect investors from fraud.

Also, in late July, Acting Comptroller of the Currency, Michael Hsu, said regulators are looking into Tether’s commercial papers to see whether each USDT token was really backed by the equivalent of one U.S. dollar.

Tether has repeatedly issued assurances that its reserves are fully backed but has yet to produce a full independent audit.

Stablecoin Ecosystem Update

Tether remains the market leader with a current supply of 69.4 billion, according to the Tether Transparency report. This is close to the all-time high for USDT, which tapped 70 billion earlier this week.

Of that total, 36 billion or 51.8% is based on the Tron network, with 33.8 billion or 48.7% running on Ethereum. USDT supply has grown by 232% since the beginning of the year.

Rival stablecoin, USDC, from Circle currently has 29.3 billion in circulation after gaining 651% in terms of supply growth so far in 2021.

SPECIAL OFFER (Sponsored)

Binance Futures 50 USDT FREE Voucher: Use this link to register & get 10% off fees and 50 USDT when trading 500 USDT (limited offer).

PrimeXBT Special Offer: Use this link to register & enter POTATO50 code to get 50% free bonus on any deposit up to 1 BTC.

You Might Also Like:


PlatoAi. Web3 Reimagined. Data Intelligence Amplified.

Click here to access.

Source: https://cryptopotato.com/u-s-treasury-targets-stablecoins-in-latest-regulatory-risk-assessment/

Continue Reading

Blockchain

Cardano, Chainlink, MATIC Price Analysis: 19 September

Published

on

Most altcoins in the market have been consolidating or recording losses over the last 24 hours. Cardano fell by 3% and inched closer to the support line of $2.20. Chainlink also depreciated by 5% and was trading closer to its three-week low price. Lastly, MATIC was seen moving closer to its one-week low price of $1.29 after registering a loss of 5% over the past day.

Cardano (ADA)

Cardano, Chainlink and MATIC Price Analysis: 19 September

ADA/USD, TradingView

Cardano lost 3% of its valuation over the last 24 hours. The altcoin was priced at $2.33. Over the last few days, ADA has been consolidating. The nearest support line for the coin stood at $2.20 and then at $1.72. 

On the four-hour 20-SMA the alt’s price was seen below it, indicating that the momentum belonged to the sellers. The Relative Strength Index was below the 50-mark. The Chaikin Money Flow also was seen below the half-line as capital inflows were low.

MACD witnessed a bearish crossover and flashed red bars on its histogram. If ADA moved on the upside, the first resistance mark stood at $2.49, toppling which it could retest $2.79. The other price ceiling stood at the multi-month high of $3.04. 

Chainlink (LINK)

Cardano, Chainlink and MATIC Price Analysis: 19 September

LINK/USD, TradingView

Chainlink was priced at $27.80 after it recorded a loss of 5% over the last 24 hours. LINK’s nearest price floor was at $27.78. Falling below which the coin could trade near its three-week low of $24.45. 

Parameters pointed towards negative price action. On the four-hour chart, LINK’s price was below the 20-SMA. This reading suggested price momentum was inclined towards the sellers. The Relative Strength Index was below the half-line.

Awesome Oscillator flashed red signal bars. MACD also displayed red bars on its histogram. On the flipside, once buying pressure revives, the altcoin could attempt to retest the $32.37 resistance mark and then revisit $35.83. 

Polygon (MATIC)

Cardano, Chainlink and MATIC Price Analysis: 19 September

MATIC/USD, TradingView

MATIC depreciated by 5% and was trading at $1.39. The altcoin’s immediate support line was at $1.29 which also is the one-week low price level. The other price floor was at its over a month-long low price point of $1.07. 

Bollinger Bands converged, indicating that price volatility would remain low over the upcoming trading sessions. MACD was bearish with red bars on its histogram. The Relative Strength Index was also seen below the half-line. 

MATIC’s movement on the upside could mean that the coin would meet with its first resistance at $1.42 and then at $1.54. Toppling over these levels, the coin could revisit its multi-month high of $1.76.

Where to Invest?

Subscribe to our newsletter

PlatoAi. Web3 Reimagined. Data Intelligence Amplified.

Click here to access.

Source: https://ambcrypto.com/cardano-chainlink-matic-price-analysis-19-september

Continue Reading

Blockchain

The Crypto Mining Fight in China Is Not Over

Published

on

It looks like China is still not done clamping down on the crypto mining space. Another region known as the Hebei province has agreed to comply with Beijing’s ruling that all crypto mining should be omitted from China’s workforce. The province is now claiming that the practice is illegal and must end within its borders no later than September 30.

China Is Still Kicking Miners Out

China shocked the world not too long ago when it decided that all crypto mining should cease. The idea was that energy used for crypto mining purposes was hazardous to the planet, and that it was setting humans on the wrong path. Thus, regulators stated that it was time to bring things to an official end.

What was most surprising about the ruling is that the country, at the time, was home to nearly 75 percent of the world’s total crypto mining operations. Thus, it stood to lose a lot of money and tax revenue by initiating the clampdown. In addition, the country is home to two of the world’s biggest developers and distributors of bitcoin mining equipment in Bitmain and Canaan Creative.

Nevertheless, China has moved forward in its decision. Many mining operators were forced to shut down their businesses and move elsewhere, and quite a few have popped up in countries such as Kazakhstan and in states like Texas and Florida. Both these regions in America have stated they are open to crypto mining projects given that they can potentially lead to healthier local and state economies, and they will create jobs for interested workers.

The Hebei province issued the following statement:

Cryptocurrency mining consumes an enormous amount of energy, which is against China’s ‘carbon neutral’ goal.

The arguments against crypto mining have become rather prominent in recent months. One of the most notable stemmed from Elon Musk, the South African entrepreneur behind billion-dollar companies such as SpaceX and Tesla. He stated early in the year that he was willing to permit bitcoin payments for electric vehicles. A few weeks later, however, he rescinded this decision, claiming that miners were not utilizing their energy correctly, and he could not condone bitcoin unless carbon emissions were brought down.

Too Much Bad Energy in the Air!

Another argument came from Kevin O’Leary of “Shark Tank” fame. The billionaire investor claimed that he would no longer be purchasing any BTC mined in China given that the country was not known to utilize green energy for mining purposes. China later took this issue to heart, it seems.

Starting in October of this year, bitcoin and crypto mining in China will be completely illegal. Regulators in the nation have stated that they will keep a close eye on the mining space and will work to punish all those who disobey the rules.

Tags: china, Crypto Mining, Hebei province
Click here for Free Trial.
Source: https://www.livebitcoinnews.com/the-crypto-mining-clampdown-in-china-is-not-over/>

Continue Reading
Uncategorized4 days ago

Wicked Craniums are now Nifty Gateway!

Blockchain3 days ago

Massive NFT and Token Giveaway from Polker as Staking is Announced!

Uncategorized4 days ago

Swissquote Confirms European Expansion Plan, Focusing on Crypto

Uncategorized4 days ago

Acorns Hires Former Amazon Executive as President, Hints at Crypto Options

Blockchain4 days ago

Biggest Crypto Adoption Rumours: Apple, Amazon, and Walmart

Blockchain4 days ago

Gate.io Introduces OpenPunks, A Community-Based NFT Collection

Uncategorized4 days ago

Investor: Coinbase’s $2 billion junk bond deal shows crypto ‘supercycle’ is in place

Uncategorized4 days ago

Head of Australian Crypto Exchange Says Regulations Are Beneficial

Blockchain4 days ago

The Signal and the Noise

Uncategorized5 days ago

How to fix the Party Member Preloading error in Apex Legends

News4 days ago

Evaluating Credit Card Debt Relief Options

Uncategorized4 days ago

Nickelodeon All-Star Brawl will include DLC fighters post-launch

Uncategorized4 days ago

Bingbon Launches its Carbon Free and Afforestation Project

News3 days ago

Gods Unchained and Guild of Guardians Layer 2 Solution Immutable Raises $60 Million

Blockchain4 days ago

Public.com Inks Deal with NFL Star to Advise on Financial Literacy Programs

Blockchain1 day ago

Over 40 days after Ethereum’s EIP-1559, here’s where it stands

Blockchain4 days ago

What’s Behind Elrond (EGLD) Daily Surges?

Uncategorized4 days ago

This needs to happen before Peter Schiff will buy Bitcoin (BTC)

Blockchain4 days ago

Ethereum’s Infura Launches Tool To Prevent Over-payment Of Fees

Blockchain4 days ago

xtingles To Drop Its First ASMR NFT “Free Like A Butterfly” On September 16

Trending