Publish/subscribe system using redis;communicate between laravel and node services

Before going deep into this ,lets discuss the typical scenerio in which we may need to implement this pattern in our backend service.

Scenerio 1: maybe for some reason,we need to run a two service for an application backend and these two separate services need a way to communicate with each other in non-blocking,asynchronous approach.

Scenerio 2 : we have to run a task that is time consuming,and also maybe resources hungry and it won't make a great user experience to perform this task in the process of user's request thereby leading to more waiting period for users to get a feedback/response for their action.
One approach to this is running a background job and another is deligating the task to separate service to process and maybe after the task is done, alert the the deligator service which we are can call Event Driven Architecture.

In one of my next articles, i will discuss how we took advantaged of the Event driven archictecture to build a Currency Trading Application,how it helped us in delivering a great realtime notifications experience and how we utilised it in our realtime chat service.

All these communication between services can still be achieved by sending http request to and fro and the use of webhook, but this won't be as effective ,non-blocking as using messaging broking approach.

There are many message broking software available like rabbitMQ ,kafka,and redis etc. but in this article ,i will be using redis.

So before starting with the implementation, lets talk about some key stuff you need to know about Pub/Sub.
Publish/subscribe is all about a scenerio where a service need to inform another service that an event has occured and also send a relating data as message to the receiver. Receiver in this case can be called the subscriber while the sender is the publisher.

The Subscriber in this case, does not need to know anything about the publishers or publisher and likewise the publisher doesn't to know anything about the subscribers, they also does not need to be related in anyway like technology stacks etc. Which means ,each service can be written in difference stacks therebby resulting into the flexibity in choice stack for a team.
The only thing connecting them together will be topic/channel in which each party is subscribing or publishing to.

Straight to implementation, all we want to do in this article is to be able to send messages/data to and fro with laravel ,and nodeJS/ExpressJS.

To setup your laravel environment and project boilerplate, please the official laravel documentations at
https://laravel.com/docs/8.x/installation.

if you are on linux based OS, you can install redis server by running the following commands in your terminal

sudo apt update
sudo apt install redis-server

sudo systemctl status redis
//to confirm the status of new redis server
//if not running ,you may need to start it by running
sudo service redis start

for windows user , you can download the zip file from https://github.com/MSOpenTech/redis/releases/download/win-3.2.100/Redis-x64-3.2.100.zip extract it and running
redis-server.exe after which you can run redis-cli.exe to open redis terminal.

for others, i will advise the use of docker.

After setting up your redis , you can confirm everything is set by running ping and if you receive back pong , we are good to go.
Now you have two option at this stage,
(I). if you are on linux based OS ,you will need to install php-redis extension using either of the commands below

sudo apt-get install php-redis

sudo apt-get install php{x.x}-redis
//where x.x is your cli php version e.g 7.4

you can also search how to install this extension on your OS,else you are going to get redis not found error.

(II). use the laravel library predis instead, start by installing the package predis/predis via composer using the command below

composer require predis/predis

after that, open the file configs/app.php in the aliases array, comment out the entry with the key Redis if already exists and replace with

'LRedis'    => Illuminate\Support\Facades\Redis::class,

After choosing either of the above options ,open the file configs/database.php and in the connections array, replace the entry with key redis with the following code block.
Note intentionally leave out the prefix config out of the configs to avoid having to add prefix to channel's name or topic later on .

'redis' => [
            'client' => 'predis',
/*if you are using the php-redis extension,you can change 'predis' here to 'phpredis'*/
            'default' => [
                'host' => env('REDIS_HOST', '127.0.0.1'),
                'password' => env('REDIS_PASSWORD', null),
                'port' => env('REDIS_PORT', 6379),
                'database' => env('REDIS_DATABASE', 0),
            ],
        ],

Now in your .env file , make sure you have something like below config in it

REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
REDIS_DATABASE=0

For this article lets assume we are a creating a food delivery appplication which can let the receiver be able to chat with the dispatcher and also able to get the real time geographical coordinate of the dispatcher's movement via websocket.
lets divide into two services ,

The Food delivery Management (We are using laravel for this),this will be the main service handling the CRUD.

The Chat and notification service (NodeJS),this will handle everything notification, chat and live location update .Basically any realtime data.

Now let prepare nodejs project ,i will be assuming you have both nodeJS and NPM installed.

create a folder for your nodejs service

mkdir {folder's name}

in the folder, init by running

npm init -y

In this article, won't be talking about about websocket or its implemention as the scope of this article is all about pub/sub.

next step is to install some other dependencies like dotenv expressjs redis redis-server if you like to manage the server from your project https://www.npmjs.com/package/redis-server and nodemon to run the script continously in development. Later in the article ,i'm going to discuss how PM2 library can be use to run our scripts continously in background and how we can manage /monitor logs.

npm i expressjs redis redis-server dotenv
npm i nodemon -g

After all is done, create a .env file in the root folder with the following as its contents.

NODE_SERVER_PORT=9016

the value will be whatever available port you will like to use.

then create a file server.js with the following as content

const http = require("http");
const express = require("express");
const app = express();
const cors = require("cors");
const redis = require("redis");
require("dotenv").config();  
const { NODE_SERVER_PORT } = process.env;
 //get port from env file

app.use(express.json());

const server = http.createServer(app);


server.listen(NODE_SERVER_PORT, function () {
    console.log("server is running.");
});

Now lets assume that ,

We've implement socket connection in our nodejs (which i will do in one of my next articles) and in this implementation, will handled everything from chat message emiting to notifications and location data.

We don't want to connect our nodeJS to the database in which laravel will be connected to. (Infact no database).

Laravel service will be the only one that can perform database operations.

Both services can be subscriber ,publisher or even both.

Assuming every party of the application are on the app(online)

So in our laravel ,when a food has been ordered by a user
we need to alert the dispatcher , one thing we do here is polling the backend(laravel service) by from the dispatcher's client for any new order, which may not be an efficient solution as polling tends to be resources intensive.
so we can connect the dispatcher's client to nodejs service using websocket.

Now that they are connected via websocket, how do we let nodejs service know that something has happened in the laravel service? yeah that's where pub/sub comes in,we can publish a message in laravel and let nodejs subscribe to the topic/channel ,so anytime nodejs receive a message from the channel ,it process it as needed.

In our laravel service, we can either publish directly from whereever we need to

//import the class
use Illuminate\Support\Facades\Redis;

/*general is the channel/topic name ,this is what our nodejs service will subscribe to
second parameter is the whatever message object we want to send encoded into json string
*/

   Redis::publish('general', json_encode(['type' => 'NewOrder','order'=>$order]));

or
even better create a NewFoodOrderedEvent by running the command php artisan make:event NewFoodOrderedEvent and
php artisan make:listener NewFoodOrderedListener --event=NewFoodOrderedEvent
and then register the event and the listener in you EventServiceProvider.php

use App\Listeners\NewFoodOrderedListener;
use App\Events\NewFoodOrderedEvent;

protected $listen = [
      ....,
        NewFoodOrderedEvent::class => [
            NewFoodOrderedListener::class  ]
,...

    ];

if you follow the event approach, you may modify your event and listener class as below.
NewFoodOrderedEvent.php

<?php

namespace App\Events;

use Illuminate\Broadcasting\Channel;
use Illuminate\Broadcasting\InteractsWithSockets;
use Illuminate\Broadcasting\PresenceChannel;
use Illuminate\Broadcasting\PrivateChannel;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Queue\SerializesModels;

class NewFoodOrderedEvent
{
    use Dispatchable, InteractsWithSockets, SerializesModels;

    /**
     * Create a new event instance.
     *
     * @return void
     */
     public $data

    public function __construct($data)
    {
        $this->data=$data
   //whatever data you passed to this event constructor
    }


}

NewFoodOrderedListener.php

<?php

namespace App\Listeners;
//import the class
use Illuminate\Support\Facades\Redis;
use App\Events\NewFoodOrderedEvent;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Queue\InteractsWithQueue;

class NewFoodOrderedListener
{
    /**
     * Create the event listener.
     *
     * @return void
     */
    public function __construct()
    {
        //
    }

    /**
     * Handle the event.
     *
     * @param  NewFoodOrderedEvent  $event
     * @return void
     */
    public function handle(NewFoodOrderedEvent $event)
    {
        //in here you can then publish to the nodeJS service
    Redis::publish('general', json_encode($event->data));
    }
}

the event can be trigger/fire as below using the event helper

$payload = ['type' => 'NewOrder','order'=>$order];
event(new NewFoodOrderedEvent($payload));

then in the nodejs, we will need to subscribe to the general channel and listen to whatever message we get

const http = require("http");
const express = require("express");
const app = express();
const cors = require("cors");
const redis = require("redis");
require("dotenv").config();  
const { NODE_SERVER_PORT } = process.env;
 //get port from env file
app.use(express.json());

//lets initialise the subscriber 
const subscriber = redis.createClient();
//subscribe to general channel

subscriber.subscribe("general");

//now listen to whatever message is sent from laravel service 
subscriber.on("message", function (channel, data) {
/*channel returns the channel's name in our case now general
and the data now will be the json string we encoded in the laravel.
This can be parse to js object using JSON.parse()

*/

}

const server = http.createServer(app);
//subscribe to general channel 


server.listen(NODE_SERVER_PORT, function () {
    console.log("server is running.");
});

NOTE: if you need to subscribe and also publish in your nodejs, you will need to initialise two separate redis client one for subscribing and the other for publishing.One intialization cant be both subscriber and publisher.

To test if everything is working as it should, run the node server with nodemon server.js and also create a simple endpoint in the laravel that send the a get request to endpoint can trigger the event.basically call the event(new NewFoodOrderedEvent($payload)); in the endpoint implementation

OR use the laravel tinker Commandline tools php artisan tinker and create a mock order object and send by calling event(new NewFoodOrderedEvent($payload)); via tinker.

Facing any error?, monitor if the laravel service is actually publish data to redis by running the redis-cli and commandline and type MONITOR to confirm.

Now, we've been able to make laravel the publisher and nodejs the subscriber.
now lets exchange role.

lets subscribe to an event from nodejs on laravel.

To this ,we need to create something that will always running just as the nodemon is continously running the server.js script.
To this , we will need to create a custom artisan command ,in which we will subscribe to redis in its implementation.
then we are going to need a process that will continously run the command so that when the event occur in nodejs, the script in laravel will be available to receive it.

We have many options we can use like nhup,supervisor,pm2 and many more.

nohup in dev is a good option but in production its not as it may stop if the system restart.
supervisor and pm2 is what i will recommend in production but i personally prefer PM2.
Enough of talks, lets create a command that will hold our subscribe code.

php artisan make:command SubscribeToGeneralChannel then in the app/console/commands folder , open the file and edit as below.

<?php

namespace App\Console\Commands;

use App\Models\Order;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Redis;

class SubscribeToGeneralChannel extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'redis:subscribe-general';
    /*
This is what will become the command we are going to use in terminal to subscribe our laravel service to nodejs
*/

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Subscribe to general channel';
/*description of the command ,this show in the laravel artisan help
*/
    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {
        parent::__construct();
    }

    /**
     * Execute the console command.
     *
     * @return int
     */
    public function handle()
    {
         //general is the name of channel to subscribe to
        Redis::subscribe(['general'], function ($message) {
             //message in here is the data strring sent/publish from nodejs 
            $messageArray = json_decode($message, true);
            //convert to php associative array
         //lets echo the message we receive from node
          echo $message;

        });
    }
}

Now lets publish from node,like i said before,to publish from node,we will need to create a separate publisher client as below,

const http = require("http");
const express = require("express");
const app = express();
const cors = require("cors");
const redis = require("redis");
require("dotenv").config();  
const { NODE_SERVER_PORT } = process.env;
 //get port from env file
app.use(express.json());
const publisher = redis.createClient();

const order = {
id:1
};
publisher.publish('general',JSON.stringify(order))'
//this will publish the order to all subscriber of the general channel which is laravel



/*//lets initialise the subscriber 
const subscriber = redis.createClient();
//subscribe to general channel

subscriber.subscribe("general");

//now listen to whatever message is sent from laravel service 
subscriber.on("message", function (channel, data) {
/*channel returns the channel's name in our case now general
and the data now will be the json string we encoded in the laravel.
This can be parse to js object using JSON.parse()

*/

}

const server = http.createServer(app);
//subscribe to general channel 


server.listen(NODE_SERVER_PORT, function () {
    console.log("server is running.");
});

This can be easily monitored by using the in-terminal of vscode so you can easily tab nodemon, and when you run php artisan redis:subscribe-general(the custom command we created earlier) ,so you can inspect the message we echoed in our custom command immplementation.

In production , pm2 can be installed by running the command
npm install pm2@latest

and starting the node server by running pm2 start server.js

and confirm its running with the command pm2 status that let you see list of all jobs.

use pm2 run the laravel subscribe command by creating a file
runlaravelsubscribetogeneralcommand.yml

nano runlaravelsubscribetogeneralcommand.yml

and write in it

apps:
  - name: runlaravelsubscribetogeneralcommand
    script: artisan
    exec_mode: fork
    interpreter: php
    instances: 1
    args:
      - redis:subscribe-general

redis:subscribe-general in the code above is the custom command we created earlier.

to start the job run the command pm2 start runlaravelsubscribetogeneralcommand.yml

to check logs of each process, you can run pm2 status

and then run pm2 logs {indexofjob} for example pm2 logs 0

in development environment ,you can run the php artisan redis:subscribe-general directly or using a longer running process nohup nohup php artisan redis:subscribe-general --daemon &.

Got any problem while following,comment below. in next article , will be discussing how this process is combined with websocket to add a realtime features to currency trading app and how we provide two clients on a channel a sync realtime timer from the backend and how the timer was kept and stopped when its neccessary.

Tags:

Pubsub Publish Subscribe Messaging

A bug that made me a better developer.

How to build powerful tables in React using the React Table package

Implementing tables using reactjs

Next.js Authentication with Netlify Identity

Using VueQuill editor in Vue.Js3

Tracing logs in GCP when exchanging message through PubSub in Java

Python e Redis: Utilizando Pub/Sub

Auto Update Site from a Git Push

Publish/subscribe system using redis;communicate between laravel and node services

Tags:

Related Posts