ZeroMq IRC Log

Friday June 3, 2011

[Time] Name	Message
[04:29] ferrouswheel	a dumb question: does zmq_poll work with inproc sockets?
[04:31] ferrouswheel	i get the error "not a socket" when I try...
[04:32] sustrik	it does
[04:33] sustrik	the error is presumably because the socket was already closed
[07:12] yukonbob	hello, #zeromq
[17:29] LazyBob	is it possible to blend classic socket applications with zero mq? As an example, say I have a datafeed from a market quote service that is done over classic sockets, and another zero mq service that's internal to my own application, is there some way I can have a select()/poll() be configured such that I can have both of them woken up? I read the docs and wasn't clear how to blend things in
[17:29] LazyBob	one thread.
[17:31] LazyBob	I have some 3rd party data feed I need to multiplex along with zeromq for ipc, but would like either events to trigger a wakeup... other option is switching to polling, which is ok, but I would like tobe able to configure between straight up polling and a blocking implementation
[17:32] mikko	LazyBob: did you check zmq_poll docs?
[17:32] LazyBob	sec
[17:33] mikko	http://api.zeromq.org/2-1:zmq-poll
[17:33] mikko	see the "Example"
[17:33] LazyBob	ok, next question:
[17:33] LazyBob	will this work in windows?
[17:34] mikko	as far as i know, yes
[17:34] mikko	but my knowledge of windows platform is very limited
[17:35] LazyBob	i tried zmq about a year ago and the windows implementation did not work for me, so I gave it some time and am ready to try it again. Maybe this time. Note, I have not tried to use this system yet, just have read the tutorials and have a general understanding. Trying to decide if it is the right solution.
[17:35] mikko	what is the problem?
[17:36] LazyBob	i'm forgetting, but back then, i tried something that wasn't supported on windows yet...
[17:36] LazyBob	i guess we will find out again . :-)
[17:36] mikko	ipc?
[17:36] mikko	thats the only thing i know that doesnt work on win
[17:36] LazyBob	yes
[17:37] LazyBob	this time around I have less of an interest in IPC and just intra-thread communications, so as long as that works I will be satisfied for this particular problem.
[18:14] ptrb	I feel like this is a really common problem that I should know the answer to, but: if I ZMQ_SUB to a socket, and then later bind another process to the same socket as ZMQ_PUB and immediately start sending, my subscriber misses the first N messages. what's the idiom to solve this?
[18:14] guido_g	wait or sync
[18:15] ptrb	(I just found the section in the manual on this, so feel free ti disregard :\|)
[18:15] guido_g	the connect is asynchronous, so if you start to send after the Ã¸mq connect there is no connection and so no messages sent
[18:21] ptrb	thanks!
[18:22] guido_g	np
[18:52] LazyBob	so I am reading this document on the zeromq site and it says that i have to have threads created in separate threads and used in those threads
[18:52] LazyBob	the question i have is... I have an ActiveX object that gets called back in these arbitrary "RPC worker threads" in Windows... I never create those threads, I just know my methods are called in those threads
[18:53] LazyBob	can I not use zmq sockets in this scenario, because the callback gets executed in random threads, is there no way to store a zmq socket member method and have it work, or am I just out of luck in this scenario
[18:54] LazyBob	in other words, am i screwed
[18:55] mikko	you have to ensure full memory barrier in order to migrate socket to another thread
[18:55] mikko	so in this case you would need to use mutex or similar exclusion mechanism
[18:56] mikko	which kinda takes away majority of the benefits
[18:57] LazyBob	so a spinlock around the socket write would still work? i.e. acquire(), zmq_send(), release() from the worker threads?
[19:02] mikko	what is the problem you are solving?
[19:04] LazyBob	so I have this trading application that listens toa bunch of stock feeds and was originally written for Linux, however now I have to use the same application but with a different broker whose API is from an ActiveX object... this ActiveX object creates all kinds of strange underlying threads, but when events are delivered, they are delivered in such a manner that callback functions are called
[19:04] LazyBob	from "RPC Worker Threads" ... I need to take the information from these callbacks, and have them "appear" to the main application as if they came over an ordinary socket.
[19:05] LazyBob	I do not have source visibility, I just know that my function can be executed within the context of some random thread somewhere... and the idea was to take structures built in the callback and pass them via inproc zmq sockets to a main loop where i/o is typically multiplexed
[19:06] LazyBob	the ActiveX object is the bane of my existence, the model is very non-standard and just flat out weird
[19:14] mikko	sounds painful
[19:16] LazyBob	it's quite awful
[19:29] cremes	LazyBob: i'm in the same situation as you
[19:29] LazyBob	very interesting
[19:29] LazyBob	same problem?
[19:29] cremes	i pull all of the data out of the COM object and enqueue it internally for my "main" thread to deal with the processing
[19:30] LazyBob	"sterling trader" happens to be the vendor, i am really hating this COM object
[19:30] cremes	in other words, i serialize all COM object access to my reactor loop (which is inherently single theaded)
[19:30] cremes	yeah, i'm using CQG
[19:30] LazyBob	they have the same problem??
[19:30] LazyBob	I couldn't maintain broker independence without this sort of single-threaded, event loop ...
[19:30] cremes	it's not a problem, per se, it's just how COM delivers events
[19:31] cremes	not sure i follow
[19:31] LazyBob	suppose I swapped out this broker for, say, Interactive Brokers
[19:31] LazyBob	or Genesis, or some other broker
[19:31] cremes	ok
[19:31] LazyBob	to have a unified model where all events are just seem in an natural state, you have to transform the events into some sort of event that works with what you call a reactor
[19:32] cremes	sure, that's what i do
[19:32] cremes	it keeps it nice and generic
[19:32] LazyBob	did you solve the problem I spoke of above with zmq?
[19:32] cremes	yes
[19:32] LazyBob	did you put synchronization around the zmq_send like Mikko suggested?
[19:32] cremes	no
[19:33] LazyBob	can you explain the model you used, so I have some idea of how to proceed/begin
[19:33] cremes	i process the data out of the COM object on whatever thread and then enqueue it for
[19:33] cremes	my reactor loop to send it out
[19:33] LazyBob	ah
[19:33] LazyBob	slightly different than me
[19:33] cremes	the reactor loop is where all sockets are created and live permanently
[20:15] GothAlice	I'm having all sorts of difficulty getting zmq installed on Mac OS X Lion. The following is the result of the compile: http://pastie.textmate.org/private/eu7oceqttu5ppbawnghx0w
[20:15] GothAlice	(Via brew.)
[20:15] GothAlice	This is the first package I've encountered with a problem. (~30 others installed fine.)
[20:16] GothAlice	If I disable PGM I have the same redeclaration failure in an io source file.
[20:31] cremes	GothAlice: very few people have access to Lion right now
[20:31] cremes	so it's hard for anyone to create a patch
[20:31] cremes	however, you have access so perhaps you could investigate it
[20:32] cremes	also, i suggest cloning the official repository and trying to build it
[20:32] cremes	perhaps this problem is specific to brew
[20:33] GothAlice	cremes: I'm in the process of doing so.
[20:33] GothAlice	I suspect is has to do with the use of LLVM exclusively in Lion.
[20:34] cremes	ah, that could be!
[20:37] mikko	GothAlice: seems to be pgm issue
[20:38] mikko	most likely < Lion doesn't define those structs
[20:38] mikko	and pgm assumes that they are not defined on darwin
[20:38] mikko	should be easy to check from ./include/pgm/in.h
[22:10] LazyBob	is it safe to assume that in most scenarios where a high water mark isn't reached, that inproc and ipc are generally lossless (with correct design)?