Friday June 3, 2011

[Time] NameMessage
[04:29] ferrouswheel a dumb question: does zmq_poll work with inproc sockets?
[04:31] ferrouswheel i get the error "not a socket" when I try...
[04:32] sustrik it does
[04:33] sustrik the error is presumably because the socket was already closed
[07:12] yukonbob hello, #zeromq
[17:29] LazyBob is it possible to blend classic socket applications with zero mq? As an example, say I have a datafeed from a market quote service that is done over classic sockets, and another zero mq service that's internal to my own application, is there some way I can have a select()/poll() be configured such that I can have both of them woken up? I read the docs and wasn't clear how to blend things in
[17:29] LazyBob one thread.
[17:31] LazyBob I have some 3rd party data feed I need to multiplex along with zeromq for ipc, but would like either events to trigger a wakeup... other option is switching to polling, which is ok, but I would like tobe able to configure between straight up polling and a blocking implementation
[17:32] mikko LazyBob: did you check zmq_poll docs?
[17:32] LazyBob sec
[17:33] mikko
[17:33] mikko see the "Example"
[17:33] LazyBob ok, next question:
[17:33] LazyBob will this work in windows?
[17:34] mikko as far as i know, yes
[17:34] mikko but my knowledge of windows platform is very limited
[17:35] LazyBob i tried zmq about a year ago and the windows implementation did not work for me, so I gave it some time and am ready to try it again. Maybe this time. Note, I have not tried to use this system yet, just have read the tutorials and have a general understanding. Trying to decide if it is the right solution.
[17:35] mikko what is the problem?
[17:36] LazyBob i'm forgetting, but back then, i tried something that wasn't supported on windows yet...
[17:36] LazyBob i guess we will find out again . :-)
[17:36] mikko ipc?
[17:36] mikko thats the only thing i know that doesnt work on win
[17:36] LazyBob yes
[17:37] LazyBob this time around I have less of an interest in IPC and just intra-thread communications, so as long as that works I will be satisfied for this particular problem.
[18:14] ptrb I feel like this is a really common problem that I should know the answer to, but: if I ZMQ_SUB to a socket, and then later bind another process to the same socket as ZMQ_PUB and immediately start sending, my subscriber misses the first N messages. what's the idiom to solve this?
[18:14] guido_g wait or sync
[18:15] ptrb (I just found the section in the manual on this, so feel free ti disregard :|)
[18:15] guido_g the connect is asynchronous, so if you start to send after the ømq connect there is no connection and so no messages sent
[18:21] ptrb thanks!
[18:22] guido_g np
[18:52] LazyBob so I am reading this document on the zeromq site and it says that i have to have threads created in separate threads and used in those threads
[18:52] LazyBob the question i have is... I have an ActiveX object that gets called back in these arbitrary "RPC worker threads" in Windows... I never create those threads, I just know my methods are called in those threads
[18:53] LazyBob can I not use zmq sockets in this scenario, because the callback gets executed in random threads, is there no way to store a zmq socket member method and have it work, or am I just out of luck in this scenario
[18:54] LazyBob in other words, am i screwed
[18:55] mikko you have to ensure full memory barrier in order to migrate socket to another thread
[18:55] mikko so in this case you would need to use mutex or similar exclusion mechanism
[18:56] mikko which kinda takes away majority of the benefits
[18:57] LazyBob so a spinlock around the socket write would still work? i.e. acquire(), zmq_send(), release() from the worker threads?
[19:02] mikko what is the problem you are solving?
[19:04] LazyBob so I have this trading application that listens toa bunch of stock feeds and was originally written for Linux, however now I have to use the same application but with a different broker whose API is from an ActiveX object... this ActiveX object creates all kinds of strange underlying threads, but when events are delivered, they are delivered in such a manner that callback functions are called
[19:04] LazyBob from "RPC Worker Threads" ... I need to take the information from these callbacks, and have them "appear" to the main application as if they came over an ordinary socket.
[19:05] LazyBob I do not have source visibility, I just know that my function can be executed within the context of some random thread somewhere... and the idea was to take structures built in the callback and pass them via inproc zmq sockets to a main loop where i/o is typically multiplexed
[19:06] LazyBob the ActiveX object is the bane of my existence, the model is very non-standard and just flat out weird
[19:14] mikko sounds painful
[19:16] LazyBob it's quite awful
[19:29] cremes LazyBob: i'm in the same situation as you
[19:29] LazyBob very interesting
[19:29] LazyBob same problem?
[19:29] cremes i pull all of the data out of the COM object and enqueue it internally for my "main" thread to deal with the processing
[19:30] LazyBob "sterling trader" happens to be the vendor, i am really hating this COM object
[19:30] cremes in other words, i serialize all COM object access to my reactor loop (which is inherently single theaded)
[19:30] cremes yeah, i'm using CQG
[19:30] LazyBob they have the same problem??
[19:30] LazyBob I couldn't maintain broker independence without this sort of single-threaded, event loop ...
[19:30] cremes it's not a problem, per se, it's just how COM delivers events
[19:31] cremes not sure i follow
[19:31] LazyBob suppose I swapped out this broker for, say, Interactive Brokers
[19:31] LazyBob or Genesis, or some other broker
[19:31] cremes ok
[19:31] LazyBob to have a unified model where all events are just seem in an natural state, you have to transform the events into some sort of event that works with what you call a reactor
[19:32] cremes sure, that's what i do
[19:32] cremes it keeps it nice and generic
[19:32] LazyBob did you solve the problem I spoke of above with zmq?
[19:32] cremes yes
[19:32] LazyBob did you put synchronization around the zmq_send like Mikko suggested?
[19:32] cremes no
[19:33] LazyBob can you explain the model you used, so I have some idea of how to proceed/begin
[19:33] cremes i process the data out of the COM object on whatever thread and then enqueue it for
[19:33] cremes my reactor loop to send it out
[19:33] LazyBob ah
[19:33] LazyBob slightly different than me
[19:33] cremes the reactor loop is where all sockets are created and live permanently
[20:15] GothAlice I'm having all sorts of difficulty getting zmq installed on Mac OS X Lion. The following is the result of the compile:
[20:15] GothAlice (Via brew.)
[20:15] GothAlice This is the first package I've encountered with a problem. (~30 others installed fine.)
[20:16] GothAlice If I disable PGM I have the same redeclaration failure in an io source file.
[20:31] cremes GothAlice: very few people have access to Lion right now
[20:31] cremes so it's hard for anyone to create a patch
[20:31] cremes however, *you* have access so perhaps you could investigate it
[20:32] cremes also, i suggest cloning the official repository and trying to build it
[20:32] cremes perhaps this problem is specific to brew
[20:33] GothAlice cremes: I'm in the process of doing so.
[20:33] GothAlice I suspect is has to do with the use of LLVM exclusively in Lion.
[20:34] cremes ah, that could be!
[20:37] mikko GothAlice: seems to be pgm issue
[20:38] mikko most likely < Lion doesn't define those structs
[20:38] mikko and pgm assumes that they are not defined on darwin
[20:38] mikko should be easy to check from ./include/pgm/in.h
[22:10] LazyBob is it safe to assume that in most scenarios where a high water mark isn't reached, that inproc and ipc are generally lossless (with correct design)?