Tech talent hiring knowledge & best practices straight to your inbox!

Faster Inter-Process Communication

Fast codechecker
A situation arised where we needed to exchange signals between a python process and a c++ one . Whenever a http request is fired to our server , a python thread handles the request . The python thread inserts the data in our MySQL table , with the column ‘processed ‘ = ‘false’ AND waits for the c++ daemon to process the data . The C++ daemon after processing the data writes to the MySQL db and also sets processed = true . The python processes takes this data and serves the request . This is how it happens .
[python highlight=”7,8,9,14″]
sql=”INSERT into tablename(data,processed)”
cursor.execute(sql); #insert into db with processed = false
processed = False
insertid = cursor.lastrowid #ID of the column we inserted
while not processed :
#loops untill the data is processed by the C++ daemon
sql= “SELECT * FROM tablename”
sql+=” where `id`=insertid and processed=`True`”
# sql to get the data once its processed
cursor.execute(sql )
row = cursor.fetchone() # executes the sql
if row != None and len(row[0]) > 0:
# TRUE if the data has been processed
processed = True
#to prevent overload on mysql server
And in the background the c++ daemon does this
[cpp hightlight=”9″]
//get any data waiting to be processed
sql=”SELECT id,data from tablename where processed =`false`”;
id,data = mysql(sql) ;
mysql(insert into tablename(data)values(data) where `id`=id);
We had a performance hit ,and while profiling we realised that mysql was getting hit very badly and the python thread was sleeping for most of the time . if we remove the time.sleep(1) in the python code , mysql was getting hit even more badly . Clearly we needed a better IPC mechanism . Yup 3rd year college linux course !! .. After some googling we landed up using named pipes and this is how we implemented it .
[python highlight=”6,7,8″]
sql=”Insert into tablename(data,processed)”
processed = False
insertid = cursor.lastrowid
os.mkfifo(insertid) #making a pipe with the name insert id
#opening the pipe for reading
#the process waits at this line for the pipe to receive some
pipe = open(insertid, “r”) data
data = pipe.readlines()[0]
#after receiving the data from
meanwhile in the background
[cpp hightlight=”9,12″]
sql=”SELECT id,data from tablename where processed =`false`”;
id,data = mysql(sql) ;
//get any data waiting to be processed
sql=insert into tablename(processed)values(“True”) where`id`=id
//set processed as true
pipe = fopen(id, “w”);
/*open the pipe for writing
notice we open the same pipe(whose name is the insertID)
on which the python process is listening for data . */
fprintf(‘pipe’,data); //write the processed data to the pipe
/*as soon as the daemon writes to the pipe
the python process knows that
the c++ daemon has processed the information*/
This way we saved our MySQL db and also made sure that our app doesnt waste time sleeping (time.sleep(1)) when a request is waiting to be served .
Take a look at how it improved our performance . Before implementing named pipes , it took around 4.2 seconds to compile and test a simple hello world program .

After implementing named pipes , we were able to bring it down to a mere 1.2 seconds 🙂

Would you like to receive similar articles straight to your inbox?