x-technology

XTechnology - Technology is beautiful, let's discover it together!

How to convert crypto currencies with GRPC microservices in Node.js

The workshop overviews key architecture principles, design patterns, and technologies used to build microservices in the Node.js stack. It covers the theory of the GRPC framework and protocol buffers mechanism, as well as techniques and specifics of building isolated services using the monorepo approach with lerna and yarn workspaces, TypeScript. The workshop includes a live practical assignment to create a currency converter application that follows microservices paradigms. It fits the best developers who want to learn and practice GRPC microservices pattern with the Node.js platform.

General

Prerequisites

Instructors

Alex Korzhikov & Andrew Reddikh

Materials

repository-open-graph-template 1

Workshop Begins!

Agenda

Introduction

Who are we?

Alex Korzhikov

alex korzhikov

Software Engineer, Netherlands

My primary interest is self development and craftsmanship. I enjoy exploring technologies, coding open source and enterprise projects, teaching, speaking and writing about programming - JavaScript, Node.js, TypeScript, Go, Java, Docker, Kubernetes, JSON Schema, DevOps, Web Components, Algorithms 👋 ⚽️ 🧑‍💻 🎧

Andrew Reddikh

andrew reddikh

Software Engineer, United Kingdom

Passionate software engineer with expertise in software development, microservice architecture, and cloud infrastructure. On daily basis, I use Node.js, TypeScript, Golang, and DevOps best practices to build a better tech world by contributing to open source projects.

What are we going to do today?

⬆️ About

Which technologies are we going to use?

⬇️ Tags

What is GRPC?

grpc logo aka pancakes

gRPC Remote Procedure Calls, of course! obvious reaction

gRPC is a modern, open source remote procedure call (RPC) framework that can run anywhere. It enables client and server applications to communicate transparently, and makes it easier to build connected systems

History

microservices graph

RPC

A distributed computing technique when

More steps involved in the process

Questions

Client 😀 ⬅️ ➡️ 💻 Server Communication

Web Protocols Or even more generic

We always need a client library to communicate to a server!

Features

Architecture

// http://protobuf-compiler.herokuapp.com/
syntax = "proto3";

package hello;

service HelloService {
  rpc JustHello (HelloRequest) returns (HelloResponse);

  rpc ServerStream(HelloRequest) returns (stream HelloResponse);

  rpc ClientStream(stream HelloRequest) returns (HelloResponse);

  rpc BothStreams(stream HelloRequest) returns (stream HelloResponse);
}

message HelloRequest {
  string greeting = 1;
}

message HelloResponse {
  string reply = 1;
}

client server communication

What are Protocol Buffers?

An efficient technology to serialize structured data

message Person {
  string name = 1;
  int32 id = 2;
  bool has_ponycopter = 3;
  // rule type name tag
  repeated uint64 vals = 4;
}

Do you know what numbers on the right side mean?

History

Features

Advanced

// rule type name tag
repeated uint64 vals = 1;

syntax = "proto3";

package hello;

service HelloService {
  rpc SayHello (HelloRequest) returns (HelloResponse);
}

message HelloRequest {
  string greeting = 1;
}

message HelloResponse {
  string reply = 1;
}

Demo

// package.json
"scripts": {
  "1. download prices": "node index.js",
  "2. generate protobuf runtime": "protoc --js_out=import_style=commonjs,binary:. prices.proto",
  "3. run protobuf transformation": "node index.js",
  "4. start grpc server": "node grpc-server.js",
  "5. start grpc client": "node grpc-client.js"
}

Demo 1 - Use protobuf to serialize and store JSON

protoc --js_out=import_style=commonjs,binary:. my.proto
# es6 not supported yet

Demo 2 - Hello GRPC Node.js server & client

// https://www.npmjs.com/package/@grpc/proto-loader#usage
const protoLoader = require('@grpc/proto-loader');
const grpcLibrary = require('grpc');
// OR
const grpcLibrary = require('@grpc/grpc-js');

protoLoader.load(protoFileName, options).then(packageDefinition => {
  const packageObject = grpcLibrary.loadPackageDefinition(packageDefinition);
});
// OR
const packageDefinition = protoLoader.loadSync(protoFileName, options);
const packageObject = grpcLibrary.loadPackageDefinition(packageDefinition);

Q&A

Crypto 🦄 Currency Converter

Prerequisites

1. Checkout demo project

Let’s get started from cloning demo monorepo

git clone git@github.com:x-technology/mono-repo-nodejs-svc-sample.git

2. Install protoc

For efficient work with .proto format, and to be able to generate TypeScript-based representation of protocol buffers we need to install protoc library.

If you’re a MacOS user and have brew package manager, the following command is the easiest way for installation:

brew install protobuf
# Ensure it's installed and the compiler version at least 3+
protoc --version

For Linux users

Run the following commands:

PROTOC_ZIP=protoc-3.14.0-linux-x86_64.zip
curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.14.0/$PROTOC_ZIP
sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc
sudo unzip -o $PROTOC_ZIP -d /usr/local 'include/*'
rm -f $PROTOC_ZIP

Alternately, manually download and install protoc from here.

3. Prepare environment

Make sure we have Node.js v14+ installed. If not, nvm is a very good tool to install multiple node versions locally and easily switch between them.

Then we need to install dependencies and bootstrap lerna within the monorepo.

yarn install
yarn lerna bootstrap

Yay! 🎉 Now we’re ready to go with the project.

Monorepo structure

For better monorepo project management we used Lerna & Yarn Workspaces

The project shapes into the following structure:

project structure

Let’s move on 🚚

Using Lerna

Lerna brings to the table few commands which can be easily executed across all/or filtered packages.

We use our common modules compiled to JavaScript, so before using it in services we need to build it first.

Following command executed build command against all common packages filtered with flag --scope=@common/*

yarn lerna run build --scope=@common/*

Common & Services

Let’s look into ./packages/common. It contains common libraries used in other places of the system. One of such libraries is @common/grpc, it contains proto generated into TypeScript/JavaScript formats as well as common gRPC server.

Following command is required to be run if we’re changing proto’s:

cd ./packages/common/go-grpc && yarn build

# OR using lerna

yarn lerna run build --scope=@common/*

For this particular task we implement only gRPC services which are stored in the folder ./packages/services/grpc. But if we decide to add rest, ./packages/common/rest is a good place to add it.

What we’re building

We’re building a currency converter, which can be used over gRPC calls.

currency convertor schema

Our intention is to send a request similar to convert 0.345 ETH to CAD and as a result we want to know the final amount in CAD and conversion rate. We also assume that, it could be more than one currency provider, e.g.

  1. Europe Central Bank rates
  2. Bank of England rates
  3. Crypto Rates

Here is how it works:

Deeper look into *.proto files

In the proto folder, according to our schema we created following files:

In implementation provider, we could just import an existing proto file and use its definitions.

import "currency-provider.proto";

package ecbProvider;

service EcbProvider {
  rpc GetRates(currencyProvider.GetRatesRequest) returns (currencyProvider.GetRatesResponse) {}
}

Let’s take a deeper look of how proto’s are generated from .proto to JavaScript.

Going back to @common/go-grpc module, we can find ./bin/build.mjs.

Here is the main command we could find there:

protoc --plugin="protoc-gen-ts=`pwd`/node_modules/.bin/protoc-gen-ts" --ts_out="service=grpc-node:`pwd`/src/proto" --proto_path="`pwd`/../../../proto/" `pwd`/../../../proto/*.proto

How to create new common lib

We use hygen for templating our new services and common libraries.

1. For example, we want to create a new logger library. 2. In the root directory run command yarn bootstrap:common and follow starter. 3. Go to the new folder in the terminal

cd ./packages/common/logger

4. Install dependencies

yarn install

5. Make sure to define appropriate name in the package.json file:

"name": "@common/logger",

Let’s follow a rule all common libraries have a prefix @common/

6. Create our library in a src/index.js

export const debug = (message: string) => console.debug(message);
export const info = (message: string) => console.info(message);
export const error = (message: string) => console.error(message);
export default { debug, info, error };

7. Make sure it builds successfully withing a command:

yarn build

8. Let’s connect our newly created library somewhere in the existing service:

yarn lerna add @common/logger --scope=@grpc/ecb-provider

9. The final step, we need to use the library inside ecb-provider service. Let’s amend file ./src/index.ts:

import logger from '@common/logger';

logger.debug('service has started');

10. Re-build ecb-provider to ensure there is no issues

yarn build

Yay! 🎉 It works!

How to create new service

1. For example, we want to create a new crypto-compare-provider service, which is another currency rate provider returning cryptocurrencies. 2. Create a folder under ./packages/services/grpc/crypto-compare-provider path. For simplicity, just copy an existing ecb-provider and rename it. 3. Go to the folder in the terminal

cd ./packages/services/grpc/crypto-compare-provider

4. Install dependencies

yarn install

5. Make sure to define appropriate name in the package.json file:

"name": "@grpc/crypto-compare-provider",

Let’s follow a rule - all grpc services have a prefix @grpc/. 6. Create a service method file packages/services/grpc/crypto-provider/src/services/getRates.ts

import { currencyProvider } from '@common/go-grpc';

export default async (
  _: currencyProvider.GetRatesRequest,
): Promise<currencyProvider.GetRatesResponse> => {
  return new currencyProvider.GetRatesResponse({
    rates: [],
    baseCurrency: 'USD',
  });
};

7. So next we need to use this method inside server.ts

import { Server, LoadProtoOptions, currencyProvider } from '@common/go-grpc';
import getRates from './services/getRates';

const { PORT = 50051 } = process.env;
const protoOptions: LoadProtoOptions = {
  path: `${__dirname}/../../../../../proto/crypto-compare-provider.proto`,
  // this value should be equvalent to the one defined in *.proto file as "package cryptoCompareProvider;"
  package: 'cryptoCompareProvider',
  // this value should be equvalent to the one defined in *.proto file as "service CryptoCompareProvider"
  service: 'CryptoCompareProvider',
};

const server = new Server(`0.0.0.0:${PORT}`, protoOptions);
server
  .addService<currencyProvider.GetRatesRequest,
    Promise<currencyProvider.GetRatesResponse>>('GetRates', getRates);
export default server;

8. Make sure it builds successfully withing a command:

yarn build

9. Start the service with the command:

yarn start

Yay! 🎉 It works!

How to test services

We use jest as a test framework and decided to write integration tests for our services. This is the best way to understand different situations, which could happen with services on the particular input/output.

1. Let’s begin from creating a test file test/services/index.spec.ts

mkdir -p test/services
touch test/services/index.spec.ts

2. First of all in the test we need to define a server, which is imported from src folder and start it in the section beforeAll

import { ecbProvider, currencyProvider, createInsecure } from '@common/go-grpc';
import server from '../../src/server';

const testServerHost = 'localhost:50061';
beforeAll(async () => {
   await server.start(testServerHost);
});
afterAll(async () => {
   await server.stop();
});

3. Next let’s create a client which will invoke server’s methods via gRPC protocol

import { ecbProvider, createInsecure } from '@common/go-grpc';

const client = new ecbProvider.EcbProviderClient(
  testServerHost,
  createInsecure(),
);

4. Let’s add first test suite in here and expect some particular result from the service’s method

describe('GetRates', () => {
 it('should return currency rates', async () => {
    const response = await client.GetRates(new currencyProvider.GetRatesRequest());

    expect(response.toObject()).toEqual({
      baseCurrency: 'EUR',
      rates: [
        { currency: 'USD', rate: 1.1348 },
      ],
    });
  });
});

5. Now it’s time to try it out with a command:

yarn test

Brilliant! 🎉 It works!

How to run this magic 🪄?

How could we run this magic to convert for us some currency?

export PORT=50052 && cd ./packages/services/grpc/ecb-provider/ && yarn start
export PORT=50051 && cd ./packages/services/grpc/currency-provider/ && yarn start

Here is a tool grpcurl for sending a test request to gRPC service from the terminal.

# list all services
grpcurl -import-path ./proto -proto ecb-provider.proto list

# list all methods of service
grpcurl -import-path ./proto -proto ecb-provider.proto list ecbProvider.EcbProvider

# call method GetRates
echo '{}' | grpcurl -plaintext -import-path ./proto -proto ecb-provider.proto -d @ 127.0.0.1:50052 ecbProvider.EcbProvider.GetRates

Hurray! 🚀

Practice

It’s time to have some practice and evolve our services even more!

Let’s grab a task based on the things you’d like to do 👇

Summary

Why?
What are the GRPC alternatives?
Compare REST RPC GraphQL
Focus Resource Action Resource
Semantics HTTP Programming Programming
Coupling Loose Tighter Loose
Format Text Binary Text

Feedback

Please share your feedback on our workshop. Thank you and have a great coding!

If you like the workshop, you can become our patron, yay! 🙏

Technologies

microservices node.js javascript protobuf grpc typescript lerna npm yarn docker git architecture crypto currency