Thursday August 26, 2010

[Time] NameMessage
[00:59] cremes mato: ok; i figured it out and posted a bunch of rambling messages to the ML
[00:59] cremes i'll add some thoughts to the wiki
[03:22] cremes btw, the pub_sub.rb example illustrates how multiple sub sockets can connect to one publisher
[03:22] andrewvc oh, yeah, I have that working with ffi-rzmq right now
[03:23] cremes that example is in zmqmachine
[03:23] cremes is your failing example similar?
[03:23] andrewvc lemme double check it
[03:23] andrewvc I actually just went with plain ffi-rzmq
[03:23] andrewvc and sent stuff off to eventmachine
[03:23] cremes ok
[03:24] andrewvc but I like zmqmachine style a bit better
[03:24] cremes yeah, i lifted most of the api from EM
[03:24] cremes except for EM::Connection which never made any sense to me API-wise
[03:25] andrewvc yeah, api wise, eventmachine is a bit convoluted
[03:25] andrewvc I see why node.js is stealing the show
[03:25] cremes ha... node.js is a javascript port of EM
[03:26] cremes the original author wrote about it on the home page way back when
[03:26] cremes part of EM's problem is due to async/callback programming is inherently ugly
[03:27] cremes it's very difficult to follow a single logical path through the code when it's broken up amongst 10s or 100s of small callbacks
[03:27] andrewvc yep
[03:27] cremes anyway, let me know if you need any bugs fixed
[03:27] andrewvc sure thing
[03:28] cremes i'm usually in channel from 8am-10pm cdt (chicago)
[03:28] cremes heading off to bed in a few...
[03:28] andrewvc thanks for the pointers to 2.0.8, reading the xrep/xreq docs right now
[03:28] cremes cool. good luck.
[03:28] andrewvc thanks
[07:31] keffo hu, asko kauppi contributed to zmq?
[07:41] sustrik some details iirc
[07:41] sustrik you know him?
[07:51] keffo never met him, but he's been around the lua community for ages
[07:52] keffo hum, xreq/xrep across say, 10 queue devices will still work as per automagic?
[07:54] guido_g it should
[07:54] guido_g as long the identity of the req is properly put in front
[07:55] keffo I got realy confused by the 'proper message sequence for xreq to xrep..." on the ml
[07:58] keffo do I need to prepend a null delimiter, or will that be taken care of? (v2.0.8)
[08:09] guido_g if you use the x* sockets, you've to add the null message yourself
[08:10] guido_g but this is only needed if you also use non x* socket (aka REQ/REP)
[08:10] guido_g so for interoperability: always add the null part
[08:11] guido_g imho, of course
[08:21] keffo ah ok
[08:22] keffo so as long as the whole flow is x-based, no additional work on my part is needed to get proper routing shebang working properly?
[08:23] guido_g shebang?
[08:25] keffo yeah... hmm, I think it's from a book
[08:26] guido_g aha
[08:26] keffo as in, the big bang resulting in the universe, ie the whole shebang :)
[08:27] keffo 1. a situation, matter, or affair (esp in the phrase the whole shebang)
[08:28] keffo anyhew, I'll have to doodle with that stuff later, right now I have to fix my server, since upgrading ubuntu always leads to something being screwed up
[08:33] guido_g hehe
[08:33] guido_g good luck then
[09:55] pieterh sustrik: why do you want to keep the iothreads argument in zmq_init()?
[09:55] pieterh it seems so wrong
[09:56] sustrik pieterh: your solution would require user to call ioclt each time
[09:57] sustrik so instead of zmq_init (1)
[09:57] pieterh ?
[09:57] keffo what's the alternative?
[09:57] pieterh in 99% of cases the user wouldn't do anything extra
[09:57] sustrik he would have to call the ioctl
[09:57] sustrik otherwise the i/o threads wouldn't be launched
[09:57] pieterh in 1% of cases they would call zmq_setctxopt() just after zmq_init()
[09:57] pieterh i/o threads can be launched automatically on first use of context
[09:57] pieterh this all seems obvious
[09:58] pieterh what am I missing?
[09:58] sustrik sync issues
[09:58] keffo why not just zmq_init( int num = 1 )? =)
[09:58] sustrik that would require mutex on each zmq call
[09:58] pieterh its the C API
[09:58] pieterh ?
[09:59] sustrik you cannot initiate I/O thread launch without locking first
[09:59] sustrik otherwise it can happen twice
[09:59] sustrik and mess everything up
[09:59] pieterh and you cannot create further i/o threads?
[09:59] sustrik no
[09:59] sustrik it's lock free, so you have to have infrastructure in place in advance
[10:00] pieterh that '1' is really nasty
[10:00] sustrik you can add one more function
[10:00] pieterh indeed
[10:01] sustrik zmq_init vs. zmq_init_thread_pool_size (int0
[10:01] pieterh zmq_init_nasty(N)
[10:01] pieterh yes
[10:01] pieterh are there other context configuration aspects that can happen dynamically?
[10:01] sustrik no
[10:02] sustrik context is a pretty dumb object
[10:02] sustrik everything is done on sockets
[10:02] pieterh indeed
[10:02] pieterh ok, this makes sense, thanks
[10:02] sustrik np
[10:02] pieterh since setctxopt was discussed several times on the list it's worth explaining why it can't happen
[10:03] pieterh i'll write this up on the 3-0 design page
[10:03] sustrik btw
[10:03] sustrik yes, there's one more possible context option
[10:03] sustrik "max_sockets"
[10:03] pieterh right
[10:03] pieterh to prevent DoS
[10:04] sustrik i mean max 0MQ sockets
[10:04] sustrik it's the same problem
[10:04] pieterh what's the use case for that?
[10:04] pieterh currently it's hard-coded?
[10:04] sustrik infrastructure for sockets has to be created in advance
[10:04] sustrik because of lock free algos
[10:04] pieterh right
[10:04] sustrik so now it's hard-coded to 512
[10:05] pieterh ok, how about an init call that is open ended
[10:05] pieterh rather than fixed arguments
[10:05] sustrik elipsis?
[10:05] pieterh an api that can be extended indefinitely
[10:05] pieterh elipsis and named arguments in C
[10:05] sustrik pretty nasty
[10:06] pieterh ZMQ_MAXSOCKETS, n, ZMQ_TTL, n, 0
[10:06] pieterh it'll be used mainly by bindings
[10:06] sustrik yeah
[10:06] pieterh and lets us add context options at any time
[10:06] sustrik possibly, let's leave that for later
[10:07] pieterh i'd like to fix all the extension points in the API by 3.0
[10:07] sustrik that's far away still
[10:08] pieterh design work takes time
[10:08] sustrik btw, the other option is to make max_sockets and io_threads compile time settings
[10:08] pieterh that's not a bad solution IMO
[10:09] pieterh but it still breaks the API so there's no point
[10:09] pieterh if we break the API it has to be for something worthwhile
[10:09] sustrik sure
[10:09] pieterh IMO once we make it easy to add context options there will be a bunch that appear
[10:10] pieterh basically it allows defaults for sockets
[10:10] pieterh e.g. modify SUB socket behavior
[10:10] pieterh set timeouts
[10:10] pieterh set heartbeats
[10:10] pieterh etc.
[10:11] pieterh these can't be compile time, they're usually per application
[10:14] CIA-20 zeromq2: 03Ivo Danihelka 07master * r43f2c6f 10/ (AUTHORS src/zmq.cpp): improved null checking in zmq_term -
[13:53] keffo hum, is there an established method of debugging how/why an xreq/xrep chain wont deliver properly?
[13:56] cremes keffo: i debugged a xreq/xrep problem by putting a queue device in the middle and having it print each message part
[13:56] cremes plus i did the same on the "client" and "server"
[13:57] keffo mm true
[13:57] keffo I already have that thougg..
[13:57] keffo though
[13:58] cremes so where does the message get stuck or disappear?
[13:58] keffo on the way back :)
[13:58] cremes ah...
[13:59] cremes how many hops are there between your client and server? are they directly connected?
[13:59] keffo client -> queue -> queue -> worker
[13:59] cremes ok
[13:59] cremes does it disappear from worker to queue1, queue1 to queue2, or queue2 to client?
[14:00] keffo the second one isn't a regular queue, it's a much hack at the moment, so the problem is probably that..
[14:00] cremes ok
[14:00] cremes are they all using XREQ/XREP sockets or have you mixed them up?
[14:00] keffo nono, made sure they're all x-based
[14:00] cremes good
[14:01] keffo but the second queue there is not using polling
[14:01] cremes that shouldn't matter
[14:01] keffo ah, it might be a multipart issue
[14:02] keffo I think I'm gonna rewrite it though, and extend the queue to handle incoming & outgoing msg callbacks instead (for complete msgs)
[14:02] keffo or does that exist somewhere in that swamp of links called wiki? =)
[14:03] cremes i don't think that exists in the "swamp" ;)
[14:03] cremes but if you write it, you should add it as a code recipe
[14:04] keffo naa, the callback would be lua :)
[14:04] cremes that's ok
[14:05] cremes i'm writing a recipe now that is all in ruby
[14:05] keffo I mean, I dont use the lua bindings
[14:05] cremes oh, so just stub it out for the recipe; everyone will have different callbacks anyway, right?
[14:15] sustrik keffo: there's one identity per hop, you have to copy all of them to the reply
[14:21] keffo sustrik, do elaborate!
[14:21] keffo before I dice up more code please! :)
[14:21] cremes keffo: i'm writing up a recipe explaining this stuff right now
[14:22] cremes give me another 15 minutes
[14:22] keffo sounds awesome
[14:22] keffo sustrik, I get what you mean, it puzzled me earlier, but I assumed zmq would pass along the identity on its own
[14:23] cremes keffo: it does but *only* for REQ and REP sockets
[14:23] guido_g not with X* sockets
[14:23] cremes XREQ and XREP make you do some of the message routing work yourself
[14:26] sustrik well, it's kind of separated into 2 layers
[14:27] sustrik the bottom layer is XREQ/XREP
[14:27] sustrik what it does is that each hop prepends the message with a new part containing the identity of the sender
[14:27] sustrik that's for requests
[14:27] sustrik for replies it works other way round
[14:28] sustrik at each hop one message part is consumer and used to route the message to the next hop
[14:28] sustrik so, when you have a look at the request after couple of hops
[14:28] sustrik what you see is a backtrace stack of identities
[14:28] sustrik followed by the original message
[14:29] sustrik as for REQ/REP
[14:29] sustrik what they do is very simple
[14:29] keffo indeed, makes sense
[14:29] sustrik REQ prepends the message by an empty message part
[14:30] sustrik so that REP socket on the other side knows where's the bottom of the stack
[14:30] keffo samples would be nice though, like a client server does >1>N hops?
[14:31] sustrik REP socket then chops the backtrace stack off the request and stores it (it knows it'll end by empty message part because it assumes there's a REQ socket on the other side)
[14:31] sustrik when the reply is sent it glues the backtrace stack at the beginning of the reply
[14:31] sustrik that's it
[14:31] sustrik keffo: what samples?
[14:32] keffo A sample that uses two queues for instance, and only using xreq/xrep
[14:32] keffo like, client>queue>worker for instance, all using x's
[14:33] keffo or rather, 2N queues, since that's what I'm doing :)
[14:33] keffo sigh.. N queues, not 2N :)
[14:33] sustrik :)
[14:33] sustrik ok, say 2 queues
[14:34] sustrik so the requester sends message X
[14:34] sustrik on first queue you'll get id1|X
[14:34] sustrik in second queue you'll get id2|id1|X
[14:34] sustrik on the terminal node you'll get id3|id2|id1|X
[14:35] sustrik then you send a reply Y like this: id3|id2|id1|Y
[14:35] keffo yup yup
[14:35] sustrik etc.
[14:35] keffo so each queue would need to keep track of all incoming, and then match with the outgoing?
[14:36] sustrik no
[14:36] sustrik all the state is in message itself
[14:36] sustrik once you forward the message you can forget about it
[14:37] keffo hum, so 2 queuedevices can be linked straight away out of the box?
[14:37] keffo or, N
[14:37] sustrik yes
[14:37] sustrik that's the whole point of the design
[14:37] sustrik it's hop-agnostic
[14:38] mankins hi. anyone around to answer a pub/sub zeromq question?
[14:39] sustrik sure
[14:39] keffo sustrik, Then I'm at a loss again why this stuff fails to deliver?
[14:39] mankins gr8. i'm trying to setup a pub/sub and am a little confused if its possible to bind+subscribe
[14:39] sustrik keffo: how do you create the reply?
[14:40] sustrik mankins: yes
[14:40] cremes keffo: take a look at this draft and let me know what youthink...
[14:40] sustrik bind/connect is completely orthogonal to pub/sub
[14:40] mankins sustrik: ok, good, I must have a problem somewhere else then. That's how I read it, but the demo didn't have that.
[14:40] keffo cremes, nice, I'll be back a bit later :)
[14:41] cremes ok
[14:41] mankins i'm also using the perl library which doesn't seem to have a poll function.
[14:43] keffo cremes, more please :)
[14:43] sustrik mankins: what's your problem?
[14:44] cremes keffo: yessir; when i'm done feel free to give specific feedback on questions that are not adequately answered
[14:44] mankins sustrik: no traffic
[14:45] mankins i'm in the middle of debugging, so not quite sure yet...when I wrap my head around it, I'll come back
[14:45] keffo cremes, Will do! Gonna take a shower, then reload the page and see?
[14:45] cremes sure
[14:47] sustrik mankins: have you subscribed?
[14:47] sustrik zmq_setsockopt (s, ZMQ_SUBSCRIBE, "", 0);
[14:47] mankins sustrik: I have two application processes, one which binds to a socket, and sets zmq_subscribe, ""
[14:47] mankins then that process just does sock->recv
[14:48] sustrik what's the socket type?
[14:48] mankins zmq_sub
[14:48] sustrik that's ok
[14:48] sustrik however, are you aware that pub/sub is unreliable?
[14:48] mankins then the other process connects to the other socket (is that the right word for an endpoint?)
[14:49] sustrik i.e. you won't get messages that were sent before you've subscribed?
[14:49] mankins it does a connect, and then send.
[14:49] mankins i'm not getting any messages, either way
[14:50] mankins actually, I think I might have got it to work by sending multiple messages.
[14:51] guido_g how so?
[14:51] mankins yeah, if I wrap the sender-side in a while loop, I can receive the messages, but if the publisher sends, then dies, the message doesn't go thru
[14:51] sustrik i would say the problem is that it takes some time to connect
[14:52] mankins ohhh. is there a way to figure out "readiness"
[14:52] sustrik nope
[14:52] mankins i see. okay...I'll work with that. thanks for your help.
[14:52] sustrik np
[15:11] mankins I'm trying to wrap an eval + alarm combo around the perl library's recv function to timeout, but this doesn't seem to be it possible the program execution is in another thread that doesn't receive the signal?
[15:14] Steve-o There was a discussion about signals before, they should be explicitly ignored so that the Perl thread should catch all of them
[15:15] mankins Steve-o: that sounds like how it should work. I wonder if the current CPAN module does that.
[15:15] Steve-o as in, ZeroMQ already explicitly "ignores" signals, as in sets them to be ignored in libc
[15:16] Steve-o if ZeroMQ is in another thread and the alarm signal is raised for the process it should bounce around till a thread doesn't ignore it
[15:17] Steve-o probably worth trying adding a signal handling inside ZMQ to see if it receives the alarm
[15:18] sustrik Steve-o: the current state of affairs is that 0MQ's internal threads block all the signals
[15:18] sustrik the application threads are untouched
[15:19] sustrik the unclear part is whether 0MQ API should return EINTR in case a signal arrives during a blocking operation
[15:19] sustrik cremes: the next is not entirely right
[15:19] Steve-o mankins: could try siggetmask() to find the state of the Perl thread
[15:19] mankins Steve-o: will look into that.
[15:19] sustrik it's only the XREP socket that adds/removes identities
[15:20] cremes ah, ok
[15:20] cremes will fix
[15:20] sustrik XREQ socket leaves the messages alone
[15:20] sustrik that way you have only a single identity in the stack per intermediary node (such as queue)
[15:20] cremes well, it *adds* its identity to each message
[15:20] cremes XREQ adds, XREP pops, yes?
[15:21] sustrik actually, XREQ does nothing
[15:21] cremes s/adds/pushes/
[15:21] sustrik XREP pushes identity of the peer to the requests
[15:21] sustrik it's a bit strange but it saves bandwidth
[15:22] cremes oh, that isn't clear from observation
[15:22] sustrik ie. in client/server topology, there's no identity passed on the wire
[15:22] cremes so XREP pushes the identity upon receipt
[15:22] sustrik yeah, you would have to inspect the traffic
[15:22] sustrik right
[15:22] cremes so who does the identity pop?
[15:22] sustrik XREP
[15:22] sustrik for replies
[15:23] cremes ok
[15:23] sustrik so it's XREP in both cases:
[15:23] sustrik push on request
[15:23] sustrik pop on reply
[15:23] cremes got it
[15:26] cremes page is fixed; please verify the part that was wrong is now either gone or correct
[15:27] mankins do I need two sockets, one for writing, and one for reading? can I change between pub and sub?
[15:27] cremes mankins: two sockets
[15:27] mankins two contexts too?
[15:27] cremes you can't change a socket type once it is allocated
[15:28] cremes mankins: maybe; i don't know what you have discussed here previously
[15:28] cremes you can have 100s or thousands of sockets per context
[15:28] cremes (actually, i think the limit is 512 right now)
[15:28] cremes per context
[15:33] sustrik cremes: the picutre still shows four identities in the message
[15:33] sustrik whereas there should be 3 as there are 3 XREP sockets
[15:33] cremes ok
[15:34] sustrik also, i would mark the process boundaries on the diagram
[15:34] sustrik that way it would be obvious you have one identity per process
[15:34] sustrik to route back
[15:35] sustrik so, in your case you have a sender process, queue1, queue2 and receiver processes
[15:35] sustrik when the request arrives at receiver
[15:35] sustrik it looks like this:
[15:36] sustrik queue2|queue1|sender|data
[15:36] cremes shouldn't it be: queue2|queue1|sender|null|data
[15:37] sustrik ah, there's a REQ socket?
[15:37] sustrik if so, yes, REQ socket adds empty message part to the bottom of the stack
[15:38] cremes no, but how else do you know when you have moved from routing parts to body parts?
[15:38] cremes it seems like a reasonable default to always use a delimiter
[15:38] sustrik but who will do that?
[15:39] cremes the "source" prepends the null part before sending the body
[15:39] sustrik yes
[15:39] cremes and source is an XREQ
[15:39] sustrik :)
[15:39] cremes heh
[15:39] cremes let me rephrase
[15:39] sustrik then the stack would look like this:
[15:40] sustrik queue2|null|queue1|null|sender|null|data
[15:40] sustrik it's the application that has to add the delimiter
[15:40] sustrik or a REQ socket
[15:40] cremes right, the app adds the delimiter
[15:40] cremes that's how i did it anyway
[15:40] sustrik ok
[15:40] keffo oh, I need to nullterminate each response?
[15:41] cremes keffo: no
[15:41] cremes sustrik thought i meant that each XREQ had to add a null
[15:41] cremes that isn't what i meant
[15:42] cremes only the source XREQ prepends the null; all queues in between just pass the message on
[15:42] cremes the destination XREP uses the null to delimit between the routing parts and the application-level body parts
[15:42] cremes make sense?
[15:43] sustrik cremes: you should start talking in terms of processes
[15:44] keffo yes
[15:44] sustrik if you say "source XREQ prepends the null"
[15:44] sustrik it sounds like 0mq does it
[15:44] keffo agreed
[15:44] cremes i'll fix that
[15:44] keffo and imo, C should be the defacto language for describing things like this?
[15:45] keffo or at the very least, pseudo :)
[15:45] cremes keffo: ruby and python are pretty close to pseudo-code already
[15:45] sustrik C is too messy
[15:45] cremes feel free to edit the page when i'm done and rewrite the code however you like ;)
[15:46] alfborge
[15:46] guido_g oh spam
[15:46] keffo cremes, If I try to rewrite code from a language I dont know, nobody will understand :)
[15:46] alfborge guido_g :)
[15:46] guido_g no joke
[15:47] alfborge sorry then.
[15:47] keffo on a completely different note, the cute little fade-in when you expand the table of contents seems to be buggy in firefox :)
[15:47] AndrewBC alfborge, I find it ironic that that page spat out a "Warning: pg_connect() unable to connect to PostgreSQL server: FATAL 1: IDENT authentication failed for user "postgres" in /var/www/database/postgres/common.php on line 10"
[15:47] AndrewBC Unless that's intentional as a joke
[15:47] alfborge AndrewBC: Actually, that's what makes it really funny.
[15:48] alfborge AndrewBC: Check the other warnings/errors.
[15:48] AndrewBC Okay, yeah. Each one does a topical error.
[15:48] AndrewBC I was too busy debating whether that was intentional to laugh. :|
[15:54] keffo cremes, Why does the page link to itself, under "see also"?
[15:55] cremes i have no idea; i can't edit that part
[15:55] cremes must be part of the page template
[15:59] pieterh keffo: that's just part of the page template
[15:59] pieterh i can fix it
[16:02] keffo ;)
[16:02] pieterh ok, that's fixed, all pages should update in a few seconds
[16:02] keffo I need to sit down and write some idiot-samples for myself to understand what I need to do here
[16:11] mankins it feels like buffers aren't getting flushed for single-message communications. is that possible?
[16:15] mankins and getting segfaults. should I be doing this in C?
[16:20] sustrik mankins: do you have a test program that causes the segfault?
[16:20] mankins i'll work on it. it seems to be random
[16:21] sustrik a sanity check: is your application single-threaded?
[16:21] mankins i just noticed that the perl docs say they are tested with zeromq 2.0.7, and I have .8 installed.
[16:21] mankins it is single threaded. i have a fcgi process and an app server
[16:21] sustrik shouldn't matter, the .8 is a stability release
[16:21] sustrik ok
[16:22] mankins at the moment I'm just trying to get them to echo
[16:23] mankins for some reason the fcgi side seems to get stuck at the recv() an strace shows it doing a recv syscall there.
[16:27] mankins i'm connecting to myself, I wonder if that's the problem.
[16:27] mankins two socks, one binds, one connects
[16:27] sustrik that should be ok
[16:27] sustrik obviously, only if you send first, connect afterwards
[16:27] sustrik recv afterwards*
[16:27] mankins it receives msgs from the client okay, but sends don't go out
[16:28] mankins i init, and setup the sockets, then go into a while loop to receive, is that fitting the model?
[16:28] sustrik sure
[16:29] mankins and topics are | delimited, right?
[16:29] mankins topic|msghere
[16:29] sustrik that's up to you
[16:29] sustrik 0mq does not care
[16:29] mankins ah
[16:29] mankins good to know.
[16:29] mankins then that really shouldn't matter.
[16:30] mankins is there a way to turn on debug to see what happens on send?
[16:30] sustrik on the wire?
[16:30] mankins y
[16:30] sustrik you may use wireshark
[16:30] sustrik to capture the traffic
[16:30] sustrik or tcpdump
[16:30] mankins ok, let me start that up. so no app-level debugging flags?
[16:31] sustrik nope
[16:40] mankins i'd listen on the TCP port, right? the transport isn't udp
[16:40] mankins nvmind
[16:44] sustrik tcp
[16:45] mankins wow, as soon as the communication starts, it doesn't stop. lots of packets.
[16:45] mankins i ran a few seconds and got 2449... but i was just trying to send a 10 byte message and receive it.
[16:46] sustrik are you sure you are not monitoring random traffic
[16:46] sustrik ?
[16:46] Steve-o like monitoring your own SSH connection isn't a good idea
[16:46] sustrik cremes: still here?
[16:47] cremes yes
[16:47] sustrik i've drawn a picture...
[16:47] sustrik wait a sec
[16:48] mankins tcpdump -s 1000 -vvv -X 'tcp port 9010' -i lo
[16:48] sd88g93 greetings
[16:48] sustrik cremes:
[16:48] sustrik does that help?
[16:49] cremes yes
[16:49] sustrik if needed, you can place it on the wiki
[16:49] sustrik [[=image x.png]]
[16:49] cremes question...
[16:50] sustrik yes?
[16:50] cremes in the queue box you have "req||X" by the down arrow
[16:50] sustrik right
[16:50] cremes where does the "req" identity come from?
[16:51] sustrik XREP socket attaches it to the message
[16:51] cremes ok, so then "X" is the body?
[16:51] sustrik yes
[16:51] cremes ok
[16:51] sustrik i should have named it body
[16:51] sustrik or data
[16:51] cremes that pic looks good
[16:52] sustrik are you on linux?
[16:52] cremes i'll add it along with a little legend so it's easier to interpret
[16:52] cremes osx
[16:52] sustrik ok
[16:52] sd88g93 is it possible to have 2 queeus set up feeding into the same socket ? or is that a bad idea ?
[16:52] sustrik i thought of sending you the source for the picture but it's a random linux drawing tool
[16:53] cremes too bad you didn't put two queues in the picture; that way i could have saved myself from rewriting the explanation
[16:53] sustrik cremes: let me do that
[16:53] cremes great
[16:53] sustrik any other changes?
[16:53] sustrik sd88g93: it's ok
[16:53] cremes make "X" into "body" and "Y" into "ResponseBody" or something similar
[16:53] sustrik the latter is too long
[16:54] cremes sure, pick something better than Y
[16:54] sustrik would stretch out of picutre
[16:54] sd88g93 sustrik: is there any special considerations to use ?
[16:54] cremes Y -> Reply
[16:54] sustrik wouldn't it be confused with the identity "rep"
[16:54] sustrik maybe change the ids to
[16:55] sustrik "app1"
[16:55] sustrik "queue"
[16:55] sustrik "app2"
[16:55] sustrik and bodies to:
[16:55] sustrik "request"
[16:55] sustrik "reply"
[16:56] sd88g93 oh tutorials added into a wiki, neat !
[16:57] cremes sustrik: that's better; any confusion in the picture can be cleared up with a legend for it
[16:58] sustrik ok, working on it
[17:01] mankins something is amiss. if the thing that binds to the socket goes down, shouldn't the tcp connection go down as well? it looks like the client re-binds the port?
[17:01] mankins or is "connect" and "bind" not related to the underlying tcp sockets?
[17:03] cremes mankins: connect & bind are not directly related to the (potentially many) TCP sockets behind the 0mq socket
[17:03] cremes take a look at the guide for a good explanation
[17:03] cremes
[17:03] mankins cremes: thanks, will read.
[17:04] sustrik cremes:
[17:04] sustrik (reload it to get rid ofcached image)
[17:05] cremes that looks *good*
[17:08] mankins i wonder if I need to do the nanosleep before another read?
[17:08] cremes how do i make the funky 0 with the line through it? what's the key combo?
[17:09] guido_g under linux its built in :)
[17:09] mankins on the mac it's alt o
[17:09] mankins ø
[17:09] cremes mankins: thanks
[17:09] mankins Ø or alt-shift-o
[17:20] Zao U+00D8
[17:56] mankins huh, looks like you can do recv(ZMQ_NOBLOCK) to do non-blocking recvs.
[17:57] sustrik :)
[18:20] mankins in case you're following along at home: I got the REQ/REP type messaging to work, but PUB/SUB would not. I haven't been able to isolate the problem, but am forging forward.
[20:23] andrew_cholakian cremes, if you're around I had a quick question about ffi-rzmq and xrep
[20:24] cremes ask away
[20:24] andrew_cholakian so, ywhen you call say copy_out_string on an xrep socket, it seems like each message comes in three parts
[20:24] andrew_cholakian the identifier, an empty string, and the actual message
[20:24] andrew_cholakian correct?
[20:25] cremes right
[20:26] cremes the identity, null msg and body are all separate ZMQ::Message objects
[20:26] andrew_cholakian oh, interesting
[20:26] cremes every time you call #recv, it returns a ZMQ::Message object
[20:26] cremes have you seen it do something else?
[20:27] andrew_cholakian no, not at all, just curious about it, because the other socket types aren't broken up like that
[20:27] cremes sure they are
[20:27] cremes for example, a SUB socket will return two message parts for each message
[20:27] cremes part 1 is the topic, part2 is the body
[20:28] cremes the bindings aren't doing anything special; they are returning exactly what the 0mq framework is generating
[20:28] cremes ZMQ::Message is really just a wrapper for the zmq_msg_* functions
[20:29] andrew_cholakian gotcha
[20:30] andrew_cholakian so,
[20:30] andrew_cholakian so is this wrong here, the way I'm doing things
[20:31] andrew_cholakian since the messages appear in a single part
[20:33] cremes they are coming in as a single message because you are only sending a topic
[20:34] cremes generally you do something like...
[20:34] cremes pub_sock.send_string("Important", ZMQ::SNDMORE)
[20:34] cremes pub_sock.send_string("Find Time Machine")
[20:34] cremes that sends two message parts which constitute a single logical message
[20:34] cremes make sense?
[20:34] andrew_cholakian makes sense
[20:35] andrew_cholakian perfect sense
[20:35] cremes good
[20:35] cremes then on the recv side you can change your handling based upon the received topic
[20:35] cremes the first msg part will contain the topic and subsequent parts contain the body of the msg
[20:35] cremes i hope that helps
[20:35] cremes feel free to add some docs! :)
[20:35] andrew_cholakian that does help actually
[20:36] andrew_cholakian I will, I'm actually working on writing documenting ruby ffi-rzmq
[20:36] cremes then you are my favorite person right now
[20:36] andrew_cholakian I'm doing a presentation to LA ruby on it in a couple weeks
[20:37] andrew_cholakian I want to get a nice collection of minimal examples for every socket class, and some example apps using a mix of em up
[20:37] cremes that sounds great
[20:37] cremes btw, you are kind of cheating in that example you pasted; sockets should be used from the same thread where their context came from
[20:37] cremes that's a 0mq requirement
[20:38] cremes the 2.1.x branch that martin is working on will allow you to move sockets between threads
[20:38] cremes unless i am misunderstanding something
[20:38] cremes perhaps sustrik can chime in
[20:38] andrew_cholakian hmmm, so I should create two contexts then
[20:38] cremes right, one per thread
[20:39] cremes er... one per application thread
[20:40] andrew_cholakian that may explain why running that example requires me to kill -9 it to leave
[20:41] cremes um.... maybe
[20:41] cremes it's probably just MRI's crazy thread handling
[20:41] cremes it works fine with jruby (no hang)
[20:41] andrew_cholakian nah, that was 1.9.2
[20:41] andrew_cholakian MRI, I think it doesn't run period
[20:42] cremes MRI = Matz Ruby Interpreter
[20:43] cremes so MRI is short-hand for 1.8.7, 1.9.1, 1.9.2 etc
[20:43] andrew_cholakian oh, I thought people usually call 1.9.2 YARV
[20:43] cremes yeah, they call it that too :)
[20:43] andrew_cholakian I'm guessing it still has the same threading issues then
[20:43] cremes 1.9.2 has broken thread handling just like 1.8.6, 1.8.7 and 1.9.1
[20:44] cremes it doesn't capture signals properly for external threads, so ctrl-c goes to never never land sometimes
[20:44] cremes and 0mq is the source of the external threads
[20:44] cremes fyi
[20:44] andrew_cholakian makes sense, yeah. I've been using killall -9 ruby quite a bit dealing with 1.9.2
[20:46] cremes i recommend jruby for use with 0mq; it has sane thread and signal handling
[20:46] cremes the main downside is it takes an extra second to start up
[20:49] andrew_cholakian yeah, I've had some issues with jruby and some stuff, I think related to eventmachine when combining them
[21:06] merimus I'm attempting to create a simple multithread pub/sub using python but am getting lots of crashes. Would anyone be kind enough to glance at my code?
[21:09] merimus starting to wonder if there isn't a bug in pythons threads
[21:13] cremes merimus: take a look at the Thread Safety section on this page:
[21:13] cremes Each ØMQ socket belonging to a particular context may only be used by the thread that created it using zmq_socket().
[21:14] cremes you should only access that socket from the thread where it was created (the main thread)
[21:17] merimus hmm... then how would I send from multiple threads?
[21:17] cremes each thread needs its own socket
[21:17] cremes let me rephrease
[21:18] cremes each thread needs to create its own socket; that socket should then only be used from that thread
[21:19] merimus but a socket is created on a particular port... which then cannot be reused. Unless I'm missing something
[21:19] cremes no, that's right
[21:19] cremes the way you get around this is to use a zmq_device
[21:19] cremes for the case of PUB/SUB, use a forwarder device
[21:20] cremes the forwarder device binds to two ports, say 5500 and 5501
[21:20] cremes all of your publisher sockets then connect to 5500
[21:20] cremes and all of your subscribers connect to 5501
[21:20] cremes the forwarder device aggregates all of the traffic from every publisher and sends to all subscribers
[21:20] cremes make sense?
[21:21] merimus indeed... heading back to the docs now
[21:21] cremes i highly recommend curling up with the guide for an hour or two:
[21:21] cremes it covers all of the basics and then some
[21:22] merimus had just started on that. Thank you very much.
[21:23] cremes you are welcome