Most efficient inter-process communication on Ubuntu


Most efficient inter-process communication on Ubuntu



I have a C++ simulation running on my server. At the same time there is C# based code which is mainly used for statistics collection. The logs that are generated by the C++ code is formatted and consumed by C# code. The two programs run on the same machine.
I refer to the simulation generating logs as "producer" and the other process which formats the logs as "consumer".



There are multiple copies of the producer that run in the system. In fact upto 20 producers can run in the system at the same time. There is just one copy of the consumer. The producer logs have unique IDs which identifies which simulation the log is coming from.



Assuming that I have large amount of memory available, what is the best method for the producers and the consumer to communicate?



At this moment each producer can generate upto 100MBps logs. So highest amount of logs that will be produced is going to be 2GBps. I suspect the consumer may not be able to absorb that much throughput and may fall behind the producers. I want to make sure that the producers are not going to get back-pressure if the consumer is not fast enough. I assume I have enough memory to absorb unprocessed logs.



I'm not sure what are my (best) options here. For the problem I have at hand, I'd like to find a flexible method of communication which is 1) Fast enough, 2) Can take care of many producer, single consumer environment, 3) Can leverage large amount of memory without needing to back-pressure the producers, 4) Provides simple APIs so that I don't have to deal with detailed of communication e.g mutexing the shared resources, etc.



Would appreciate any advice.




1 Answer
1



So, identify how you will implement it, find some examples of shared memory solutions vs. message passing, and try! Get it right, then you can make a more informed decision on which is better for your particular use-case.



I suppose others are not quick to answer your question since there is just not one method of interprocess communication that is superior to others. Especially when the question is asked in isolation - with no larger context of what the larger application is for. What are all these logs for?





Thanks for the insight. I added the important info you mentioned in the original question I asked.
– Amir
yesterday







By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Comments

Post a Comment

Popular posts from this blog

paramiko-expect timeout is happening after executing the command

paramiko-expect timeout is happening after executing the command Can someone help me why timeout is happening after executing the command .I am trying to SSH to a machine and execute a command.I want to store the result of the executed command . I have referred paramiko-timeout but i didn't get my expected result. import traceback try: import paramiko except ImportError: raise Exception("Please install paramiko , pip install paramiko") import pexpect from paramiko_expect import SSHClientInteraction def main(): HOSTNAME = "minion5.net" USERNAME = "root" PASSWORD = "help@432" PROMPT = "root@*:~$s+" try: client = paramiko.SSHClient() client.load_system_host_keys() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(hostname=HOSTNAME, username=USERNAME, password=PASSWORD) with SSHClientInteraction(client, timeout=20, display=True) as interact: ...
Read more

Opening a url is failing in Swift

Opening a url is failing in Swift breakpointing inside the callback and logging success shows false. I added itms-apps to my plist: LSApplicationQueriesSchemes and put this in my app delegate: func application(_ app: UIApplication, open url: URL, options: [UIApplicationOpenURLOptionsKey : Any] = [:]) -> Bool { return true } func moreTapped() { let url = NSURL(string: "itms-apps://itunes.com/developer/quantum-productions/id979315877") if #available(iOS 10.0, *) { UIApplication.shared.open(url! as URL, options: [:], completionHandler: {(success: Bool) in }) } else { UIApplication.shared.openURL(url! as URL) } } Are you testing on a real device? – rmaddy Jun 21 at 23:57 I try on a device, get "Cannot Connect to App Store". Device is online ...
Read more

Export result set on Dbeaver to CSV

Export result set on Dbeaver to CSV Normally I use Dbeaver for windows and always export my result set like this: run my query --> select the result --> export the result set --> select export to clipboard --> done This step by step puts my result set in my clipboard and I can paste it wherever I want to work with it. The problem is that now I am using dbeaver for mac and this guide is not working. I can go on until the moment that I select my result set like in the image below: exporting data set But once I go further in the process, in the last step I get: no query Note that in "source" it was suppose to show the query that originated the result set, but instead it says just "select. As a result it does't select my result or anything (besides being "successful"). Normally my query would show up there automatically and I couldn't find any option that corrects this problem in the menus. Please help me! Thanks. Aft...
Read more