[Time] Name | Message |
[00:24] sd88g93
|
ok, fixed one problem, but still doesnt work with the forwarder and the threading: http://pastebin.com/7wB68Hjy
|
[00:24] sd88g93
|
even just one thread
|
[02:05] fruminator
|
hey all, I sent a post to the mailing list but I don't see it in the archives. did it get through?
|
[02:09] sd88g93
|
just got it , frum
|
[02:09] sd88g93
|
i'm having a similar problem, but i'm using the forwarder programmatically , the zmq_device() function
|
[02:11] sd88g93
|
fruminator: in my case, i'm trying to publish a messsage from multiple threads at one socket point, using inproc communication, i can only get it to work when i dispense with the forwarder device and use it in one thread
|
[02:11] sd88g93
|
i have the forwarder device to forward from each thread to the main socket in the main thread
|
[02:12] sd88g93
|
there's not much documentation on the forwarder,
|
[02:13] sd88g93
|
http://pastebin.com/7wB68Hjy
|
[02:14] sd88g93
|
fruminator: did you try using "tcp://lo:5555" instead of 127.0.0.1 ?
|
[02:16] fruminator
|
I will try that. I have to use the forwarder because this is between 2 totally different systems
|
[02:18] sd88g93
|
all it is is an infinite loop that recieves at one end and sends on the other
|
[02:18] fruminator
|
dont follow
|
[02:18] sd88g93
|
ok lol
|
[02:18] fruminator
|
what is?
|
[02:20] sd88g93
|
actually, i can get it to work if i keep it to one thread doing the publishing, I will probably have to code my own queuer , to queue the messages up before publishing
|
[02:20] fruminator
|
seems like a different use case than I'm facing.
|
[02:21] fruminator
|
the out of the box behavior is perfect for me; I just need it to work offline
|
[02:21] sd88g93
|
oh ok
|
[02:23] fruminator
|
gotta go for now, hope to see some replies on my email. thanks!
|
[05:27] ak47
|
I'm having a bit of a problem with ZMQ_RCVMORE
|
[08:51] jugg
|
sustrik: it looks like you've removed issue support on your zeromq2 github page. How would you like patches submitted to you?
|
[08:53] jugg
|
anyway, here is a patch that adds ZMQ_EVENTS to zmq_getsockopt() -> http://paste2.org/p/916561
|
[08:58] feroz
|
Hello !
|
[09:00] jugg
|
sustrik: in any case with your latest code base zmq_term never returns.
|
[09:03] feroz
|
I'm reading some docs about 0mq, and looking into the Pub/Sub pattern. From examples i see, web subscribing to a channel, you only match against the begining of a string, is there anyway to do more ?
|
[09:08] guido_g
|
feroz: no
|
[09:14] feroz
|
Okay, do you think it would be easy to extend that from actual source code?
|
[09:15] guido_g
|
you mean as a patch to ÃMQ itself?
|
[09:26] jugg
|
feroz: what "more" do you want to do?
|
[10:10] feroz
|
Im thinking of serializing objects and send them thought Pub/Sub
|
[10:10] feroz
|
So it could be nice if i could filter against some attribute
|
[10:14] jugg
|
just filter it yourself, post recv() then.
|
[10:15] jugg
|
I suppose the subscription interface could be modified to be able to register a callback for custom publication filtering, but in your case the entire message content would have to be passed along so it could be unserialized, at which point you've gained nothing.
|
[10:18] feroz
|
Okay, thanks !
|
[10:49] sustrik
|
jugg: yes, I am aware of that
|
[10:49] sustrik
|
sustrik/zeromq2 happens to have shutdown broken
|
[10:50] jugg
|
ok
|
[10:52] sustrik
|
jugg: what was that about the erlang binding?
|
[10:54] jugg
|
Well, I have it compiling, and it can send data, but when I receive that data in a non-erlang app, it appears to be corrupted.
|
[10:54] jugg
|
I have not yet been able to get the erlang recv functionality to work either.
|
[10:54] jugg
|
it just blocks
|
[10:55] sustrik
|
i see
|
[10:57] jugg
|
not sure if it is interesting or not, but the general term() blocking issue can be reproduced with (lua code): http://paste2.org/p/916658 - remove the connect line: no issue, change REQ to REP: no issue, change connect to bind: no issue.
|
[10:58] jugg
|
however, if I have both a REQ and a REP and the send/recv to each other just one, both will lock on term()
|
[10:58] jugg
|
s/just one/just once/
|
[11:03] sustrik
|
jugg: yes, the term code got broken during work on socket migration between threads
|
[11:04] sustrik
|
i have to fix it
|
[11:05] jugg
|
yah. I'm just trying to reduce the known problems while debugging this erlang binding and ran across that termination issue while testing outside of erlang.
|
[11:07] jugg
|
I realize these issues are from your socket migration work - so just fyi in case you hadn't noticed it yet. Are there any other major issues you've found from the migration work?
|
[11:19] sustrik
|
no, i don't think so
|
[11:19] sustrik
|
shutdown is the only problem i am aware of
|
[11:22] jugg
|
sustrik: on the erlang bindings, I'm not getting this ZMQ_FD thing. It get the fd, when it signals, I can use ZMQ_EVENTS, however from my reading of has_in/out, those won't return true until socket's process_commands() is called - in which case, it would seem a mechanism is needed to call that.
|
[11:26] jugg
|
s/It get the fd/I get the fd and wait on it/
|
[11:32] sustrik
|
jugg: it can be called from getsockopt IMO
|
[11:32] sustrik
|
ZMQ_EVENTS
|
[11:33] jugg
|
Is that correct that it needs to be called then?
|
[11:34] sustrik
|
yes
|
[11:34] sustrik
|
process_commands gets the socket state up to date
|
[11:35] sustrik
|
if it is not called, getsockopt would report some historical state
|
[11:38] jugg
|
yay, that fixed the bindings.
|
[11:39] jugg
|
Now to figure out why sending messages external to erlang appears to corrupt them.
|
[11:41] jugg
|
sustrik: here is an updated ZMQ_EVENTS then -> http://paste2.org/p/916697
|
[11:43] jugg
|
sustrik: what happens if pending events are not handled, and the fd is waited upon again?
|
[11:45] sustrik
|
it'll return immediately
|
[11:48] sustrik
|
jugg: your patch looks ok
|
[11:48] sustrik
|
would you like to contribute it?
|
[11:50] jugg
|
It was originally written by either Dhammika or Serge who wrote the bindings... I only moved it to socket_base and cleaned it up...
|
[11:51] jugg
|
hmm, actually looking at their original code, I've fairly well redone it... so yes, I'll contribute it.
|
[11:52] sustrik
|
thanks
|
[11:52] sustrik
|
can you send the patch to the mailing list saying it's submitted under MIT license?
|
[11:52] jugg
|
ok
|
[11:58] feroz
|
hey, is there any paper explaining how pub/sub was implemented in zeromq?
|
[12:00] Lazesharp
|
hi guys, are there any docs/articles on streamer devices?
|
[12:00] jugg
|
feroz: not sure exactly what you mean, but: http://api.zeromq.org/zmq_socket.html#_publish_subscribe_pattern
|
[12:01] feroz
|
Thanks jugg
|
[12:05] sustrik
|
Lazesharp: there's none
|
[12:05] jugg
|
sustrik: Would you like ZMQ_EVENTS to be stubbed out in the doc/zmq_getsockopt.txt file with this patch as well, or keep doc update in a separate patch?
|
[12:05] sustrik
|
but it's pretty obvious
|
[12:06] sustrik
|
it gets messages from an UPSTREAM sockets and passes them to a DOWNSTREAM socket
|
[12:06] sustrik
|
jugg: single patch
|
[12:07] Lazesharp
|
sustrik: any queuing if there are no downstream sockets connected?
|
[12:07] Lazesharp
|
or are messages just discarded
|
[12:08] Lazesharp
|
actually, these are just building blocks - there's nothing stopping me writing my own "queueing streamer" is there
|
[12:11] sustrik
|
yes, it queues messages
|
[12:12] Lazesharp
|
oh right, awesome
|
[12:13] jugg
|
sustrik: is the document line wrap at 78 chars?
|
[12:23] jugg
|
sustrik: this is the text I came up with. Sufficient for the patch? http://paste2.org/p/916725
|
[12:24] sustrik
|
80 chars
|
[12:26] jugg
|
it should probably mention some relation to ZMQ_FD... but that isn't documented yet.
|
[12:27] sustrik
|
i'll do that
|
[12:27] sustrik
|
there's an in parameter to getsockopt (ZMQ_EVENTS)
|
[12:27] sustrik
|
?
|
[12:27] sustrik
|
that's kind of strange
|
[12:28] sustrik
|
i would say it should check both IN & OUT
|
[12:28] jugg
|
oh?
|
[12:29] sustrik
|
once the events were processed (process_commands), actual has_in & has_out are pretty lightweight, so there's no real performance penalty
|
[12:30] sustrik
|
it just seems wrong the getsockopt would accept an in parameter
|
[12:31] jugg
|
ok, it was a carry over from how it was being used in the erlang binding. I can change the binding's use of it of course.
|
[12:32] jugg
|
I'll update the patch then.
|
[12:33] sustrik
|
thx
|
[12:35] jugg
|
ok, this is what I have now: http://paste2.org/p/916737
|
[12:42] sustrik
|
nice & simple
|
[12:42] sustrik
|
post it to the ml and I'll patch the codebase
|
[12:43] jugg
|
sent
|
[12:45] sustrik
|
thx
|
[14:19] jugg
|
meh, these erlang bindings are worthless for sending messages with a non erlang application. The bindings packs the data so that the receiving end must also be an erlang application to unpack them.
|
[14:20] jugg
|
what point was it to ever create these bindings except for the use case of interacting with a non erlang application?!
|
[16:49] erickt
|
good morning #zeromq. question about up/downstreaming sockets. I was playing around with my own version of the butterfly example, and I found that when I killed one of the parallel workers, the other workers didn't pick up the leftover work. Is there a common pattern for dealing with problems like this?
|
[16:54] dirtmcgirt
|
i've got a PUB sending at a fast rate over a IPC socket, but messages aren't returned by zmq_recv in the SUB until the PUB process exits
|
[17:03] sustrik
|
dirtmcgirt: do you have a test program?
|
[17:03] dirtmcgirt
|
sustrik: i'll see if i can write one up
|
[17:04] sustrik
|
good, please report the issue using the bug tracker then
|
[17:04] dirtmcgirt
|
noticed it in production last night - the memory on the SUB process swelled, but zmq_recv didn't deliver
|
[17:04] dirtmcgirt
|
is this something that's been seen, or new?
|
[17:04] sustrik
|
no, i haven't seen that yet
|
[17:04] cremes
|
erickt: just a guess, but i think the upstream socket you killed had a bunch of "work" messages queued up
|
[17:05] erickt
|
yeah, that's what I was thinking
|
[17:05] cremes
|
erickt: you might want to set the HWM for the upstream socket to 1 so the downstream socket knows not to send it more work than it can handle before dying
|
[17:05] cremes
|
or so you don't lose work msgs
|
[17:06] erickt
|
oh that's a good idea
|
[17:10] erickt
|
oh, but if the component had already started working, there's still a chance it could die halfway through processing that message. I guess in that case it'd make sense to have another socket to the upstream node to send back heartbeat status updates
|
[17:10] erickt
|
or, just log the state in some database
|
[18:36] cremes
|
erickt: agreed; PUB and DOWNSTREAM are fire-and-forget so if you need guaranteed delivery you need to build that on top
|
[19:08] erickt
|
thanks cremes
|
[19:36] cremes
|
you are welcome
|
[20:20] erickt
|
does zeromq support dns-sd yet? I've seen a couple emails mentioning using that for service discovery
|
[20:20] erickt
|
support directly I mean, as opposed to doing the dns queries myself
|
[20:35] sustrik
|
erickt: not yet
|
[20:36] sustrik
|
it's a research issue
|
[20:36] erickt
|
thanks again
|
[21:39] erickt
|
I'm sorry to keep asking questions, but is there a public roadmap? Or if not, is there an estimated time when the failover for streaming or REP/REQ will be implemented?
|