Last week, my team finished developing and submitting our MarketMake 2021 product FidoByte — a tokenized shelter-dog walking protocol. One of the most difficult parts of this project was figuring out how to manage a flexible database both on and off chain.
In this article, I want to show how data flows through our project and how you can replicate the same architecture.
Here are the three steps we will cover
- MongoDB Atlas and Realm
- Chainlink External Adapter and Node Job Spec
- Oracle Contract and Decoding Bytes
MongoDB Atlas and Realm: A Fully Managed GraphQL API Solution
The randomly generated dog walking data is here, with 500 walks from 55 unique walkers. MongoDB Atlas is easy to navigate, just set up a project and create a collection. Then, take your data and insert it as JSON to fill your database. If you need to go from CSV to JSON easily, you can check out this site.
After this, click on Realm and add this collection (in my case the “walks” collection) to a new project. The UI will take you through generating a GraphQL schema and endpoint that you can write POST requests to. Next, you will want to go to “Authentication Providers”, turn on “Email/Password” and create a login. To test this out, you can use Postman or Curl like with the following query:
There are many other authentication options as well, and if you’re using a frontend to access this API then check out their Task Tracker tutorial detailing how to use their Realm App npm package. If you just want to query and don’t need the ability to set up accounts in your frontend, all you need is this before your query:
const app = new Realm.App("petproject-sfwui");await app.logIn(Realm.Credentials.emailPassword("[email protected]", "test123"));
Now you should be able to query to your hearts content (for the first 1M requests at least)!
Creating your Chainlink External Adapter and accompanying Job Specification
Okay, so now we have an authenticated endpoint for our API — but this makes getting data on-chain a little more difficult. If you just want to make a GET request to a REST API, then you can use the Chainlink GET adapter with any job specification and just pass the URL and copyPath that you want.
However given this is a secure, multi-line GraphQL query and the data had to be aggregated for my smart contracts (i.e. total distance walked by Ethereum address), I needed to create a Chainlink external adapter. Luckily, Chainlink has provided an EA template where you only need to change the index.js file and can easily spin up a localhost:8080 server to test your queries to. For a detailed guide, check out Patrick’s openweather external adapter walkthrough here or follow along with my adapter repo. If you’ve been Googling around, you’re probably here because you want to figure out how to pass parameters into the query, how to format and manage your responses, and how to write your job specification.
For your query, you can place what you need into customParams . Make sure you name your variable something that is not reserved, for example if you want to pass in an address call your variable input_address otherwise you will run into issues later as address will return the oracle address instead of your query.
For formatting your response, you can use Requester.Success(...) to automatically format your output the way the Chainlink Node needs to copy to Ethereum, or you can return like this:
Also, there are limited supported types for how you can return your data to your smart contract. ethuint256 is the most popular one, and currently ethuint256[] arrays are not supported. I had four variables I needed to return, so I used a reduce to concatenate them in a string with a padding of 6.
It’s okay to return a string here, because it will be encoded to an uint256 right before it is sent to your contract. That brings us to the job specification:
Let’s break this down a bit. runLog is the initiator, meaning this will only be run when a request is logged in Ethereum. The tasks are run in order: first with dog_walking_EA adapter, then the result path from datais copied to the node, then it is encoded from a string to an ethuint256 variable, and lastly it is sent back to the requester/contract with an ethtx that is visible on chain.
I was lucky enough to find a node operator to host my EA on their Bridge and my Job Spec as well, if you need to find a similarly helpful soul check out the #node-operator-requests channel on the Chainlink Discord.
Creating your Oracle Contract to make your query and decoding the return data
You’re so close to being done now! Lastly comes the Oracle contract. There are three key items here too:
- Make sure you have @chainlink/contracts installed, and import “@chainlink/contracts/src/v0.6/ChainlinkClient.sol”
- It’s up to you if you want to create functions to set the jobID, Oracle Address (Node Address), and Fee (typically 0.1 LINK); or to just set it in the constructor. For sake of simplicity in this project, I set mine all in the constructor. Make sure you have
setPublicChainlinkToken()in there otherwise your request will NOT go through.
3. You will have to get familiar with changing variables of different types into strings, since that is the only thing you can put when adding a parameter to a request. So in my case I wanted to input an address, and created a library with a bunch of type to type functions, including anaddressToString(address) function. To parse the uint256 return into different variables, I used a getSliceInt(uint,start,end) function. You can find my library here if you need it (typesLibrary.sol).
And there you have it! We can now use this request to get the total distance walked, time walked, dogs walk, and amount that should be paid from our original dataset.
In Conclusion
Oracles and off-chain data can get confusing, especially when it then comes to decisions on how to responsibly manage and decentralize these feeds. While we are still in the early stages of figuring out what should and shouldn’t be stored in smart contracts (due to speed, storage, cost, privacy, and other considerations), there is no doubt that this will become more and more complex in the future. Knowing how to create a Chainlink External Adapter to manage complex queries and authentications is a must have skill in Web3 development, and I hope this guide has helped clear any blockers in your way. Feel free to reach out to me if you have any lingering questions, I’ll be with other developers in the Chainlink Discord.
Join Coinmonks Telegram group and learn about crypto trading and investing
