IRC Log

Thursday August 26, 2010

[Time] NameMessage
[00:59] cremes mato: ok; i figured it out and posted a bunch of rambling messages to the ML
[00:59] cremes i'll add some thoughts to the wiki
[03:22] cremes btw, the pub_sub.rb example illustrates how multiple sub sockets can connect to one publisher
[03:22] andrewvc oh, yeah, I have that working with ffi-rzmq right now
[03:23] cremes that example is in zmqmachine
[03:23] cremes is your failing example similar?
[03:23] andrewvc lemme double check it
[03:23] andrewvc I actually just went with plain ffi-rzmq
[03:23] andrewvc and sent stuff off to eventmachine
[03:23] cremes ok
[03:24] andrewvc but I like zmqmachine style a bit better
[03:24] cremes yeah, i lifted most of the api from EM
[03:24] cremes except for EM::Connection which never made any sense to me API-wise
[03:25] andrewvc yeah, api wise, eventmachine is a bit convoluted
[03:25] andrewvc I see why node.js is stealing the show
[03:25] cremes ha... node.js is a javascript port of EM
[03:26] cremes the original author wrote about it on the home page way back when
[03:26] cremes part of EM's problem is due to async/callback programming is inherently ugly
[03:27] cremes it's very difficult to follow a single logical path through the code when it's broken up amongst 10s or 100s of small callbacks
[03:27] andrewvc yep
[03:27] cremes anyway, let me know if you need any bugs fixed
[03:27] andrewvc sure thing
[03:28] cremes i'm usually in channel from 8am-10pm cdt (chicago)
[03:28] cremes heading off to bed in a few...
[03:28] andrewvc thanks for the pointers to 2.0.8, reading the xrep/xreq docs right now
[03:28] cremes cool. good luck.
[03:28] andrewvc thanks
[07:31] keffo hu, asko kauppi contributed to zmq?
[07:41] sustrik some details iirc
[07:41] sustrik you know him?
[07:51] keffo never met him, but he's been around the lua community for ages
[07:52] keffo hum, xreq/xrep across say, 10 queue devices will still work as per automagic?
[07:54] guido_g it should
[07:54] guido_g as long the identity of the req is properly put in front
[07:55] keffo I got realy confused by the 'proper message sequence for xreq to xrep..." on the ml
[07:58] keffo do I need to prepend a null delimiter, or will that be taken care of? (v2.0.8)
[08:09] guido_g if you use the x* sockets, you've to add the null message yourself
[08:10] guido_g but this is only needed if you also use non x* socket (aka REQ/REP)
[08:10] guido_g so for interoperability: always add the null part
[08:11] guido_g imho, of course
[08:21] keffo ah ok
[08:22] keffo so as long as the whole flow is x-based, no additional work on my part is needed to get proper routing shebang working properly?
[08:23] guido_g shebang?
[08:25] keffo yeah... hmm, I think it's from a book
[08:26] guido_g aha
[08:26] keffo as in, the big bang resulting in the universe, ie the whole shebang :)
[08:27] keffo 1. a situation, matter, or affair (esp in the phrase the whole shebang)
[08:28] keffo anyhew, I'll have to doodle with that stuff later, right now I have to fix my server, since upgrading ubuntu always leads to something being screwed up
[08:33] guido_g hehe
[08:33] guido_g good luck then
[09:55] pieterh sustrik: why do you want to keep the iothreads argument in zmq_init()?
[09:55] pieterh it seems so wrong
[09:56] sustrik pieterh: your solution would require user to call ioclt each time
[09:57] sustrik so instead of zmq_init (1)
[09:57] pieterh ?
[09:57] keffo what's the alternative?
[09:57] pieterh in 99% of cases the user wouldn't do anything extra
[09:57] sustrik he would have to call the ioctl
[09:57] sustrik otherwise the i/o threads wouldn't be launched
[09:57] pieterh in 1% of cases they would call zmq_setctxopt() just after zmq_init()
[09:57] pieterh i/o threads can be launched automatically on first use of context
[09:57] pieterh this all seems obvious
[09:58] pieterh what am I missing?
[09:58] sustrik sync issues
[09:58] keffo why not just zmq_init( int num = 1 )? =)
[09:58] sustrik that would require mutex on each zmq call
[09:58] pieterh its the C API
[09:58] pieterh ?
[09:59] sustrik you cannot initiate I/O thread launch without locking first
[09:59] sustrik otherwise it can happen twice
[09:59] sustrik and mess everything up
[09:59] pieterh and you cannot create further i/o threads?
[09:59] sustrik no
[09:59] sustrik it's lock free, so you have to have infrastructure in place in advance
[10:00] pieterh that '1' is really nasty
[10:00] sustrik you can add one more function
[10:00] pieterh indeed
[10:01] sustrik zmq_init vs. zmq_init_thread_pool_size (int0
[10:01] pieterh zmq_init_nasty(N)
[10:01] pieterh yes
[10:01] pieterh are there other context configuration aspects that can happen dynamically?
[10:01] sustrik no
[10:02] sustrik context is a pretty dumb object
[10:02] sustrik everything is done on sockets
[10:02] pieterh indeed
[10:02] pieterh ok, this makes sense, thanks
[10:02] sustrik np
[10:02] pieterh since setctxopt was discussed several times on the list it's worth explaining why it can't happen
[10:03] pieterh i'll write this up on the 3-0 design page
[10:03] sustrik btw
[10:03] sustrik yes, there's one more possible context option
[10:03] sustrik "max_sockets"
[10:03] pieterh right
[10:03] pieterh to prevent DoS
[10:04] sustrik i mean max 0MQ sockets
[10:04] sustrik it's the same problem
[10:04] pieterh what's the use case for that?
[10:04] pieterh currently it's hard-coded?
[10:04] sustrik infrastructure for sockets has to be created in advance
[10:04] sustrik because of lock free algos
[10:04] pieterh right
[10:04] sustrik so now it's hard-coded to 512
[10:05] pieterh ok, how about an init call that is open ended
[10:05] pieterh rather than fixed arguments
[10:05] sustrik elipsis?
[10:05] pieterh an api that can be extended indefinitely
[10:05] pieterh elipsis and named arguments in C
[10:05] sustrik pretty nasty
[10:06] pieterh ZMQ_MAXSOCKETS, n, ZMQ_TTL, n, 0
[10:06] pieterh it'll be used mainly by bindings
[10:06] sustrik yeah
[10:06] pieterh and lets us add context options at any time
[10:06] sustrik possibly, let's leave that for later
[10:07] pieterh i'd like to fix all the extension points in the API by 3.0
[10:07] sustrik that's far away still
[10:08] pieterh design work takes time
[10:08] sustrik btw, the other option is to make max_sockets and io_threads compile time settings
[10:08] pieterh that's not a bad solution IMO
[10:09] pieterh but it still breaks the API so there's no point
[10:09] pieterh if we break the API it has to be for something worthwhile
[10:09] sustrik sure
[10:09] pieterh IMO once we make it easy to add context options there will be a bunch that appear
[10:10] pieterh basically it allows defaults for sockets
[10:10] pieterh e.g. modify SUB socket behavior
[10:10] pieterh set timeouts
[10:10] pieterh set heartbeats
[10:10] pieterh etc.
[10:11] pieterh these can't be compile time, they're usually per application
[10:14] CIA-20 zeromq2: 03Ivo Danihelka 07master * r43f2c6f 10/ (AUTHORS src/zmq.cpp): improved null checking in zmq_term - http://bit.ly/afqWj9
[13:53] keffo hum, is there an established method of debugging how/why an xreq/xrep chain wont deliver properly?
[13:56] cremes keffo: i debugged a xreq/xrep problem by putting a queue device in the middle and having it print each message part
[13:56] cremes plus i did the same on the "client" and "server"
[13:57] keffo mm true
[13:57] keffo I already have that thougg..
[13:57] keffo though
[13:58] cremes so where does the message get stuck or disappear?
[13:58] keffo on the way back :)
[13:58] cremes ah...
[13:59] cremes how many hops are there between your client and server? are they directly connected?
[13:59] keffo client -> queue -> queue -> worker
[13:59] cremes ok
[13:59] cremes does it disappear from worker to queue1, queue1 to queue2, or queue2 to client?
[14:00] keffo the second one isn't a regular queue, it's a much hack at the moment, so the problem is probably that..
[14:00] cremes ok
[14:00] cremes are they all using XREQ/XREP sockets or have you mixed them up?
[14:00] keffo nono, made sure they're all x-based
[14:00] cremes good
[14:01] keffo but the second queue there is not using polling
[14:01] cremes that shouldn't matter
[14:01] keffo ah, it might be a multipart issue
[14:02] keffo I think I'm gonna rewrite it though, and extend the queue to handle incoming & outgoing msg callbacks instead (for complete msgs)
[14:02] keffo or does that exist somewhere in that swamp of links called wiki? =)
[14:03] cremes i don't think that exists in the "swamp" ;)
[14:03] cremes but if you write it, you should add it as a code recipe
[14:04] keffo naa, the callback would be lua :)
[14:04] cremes that's ok
[14:05] cremes i'm writing a recipe now that is all in ruby
[14:05] keffo I mean, I dont use the lua bindings
[14:05] cremes oh, so just stub it out for the recipe; everyone will have different callbacks anyway, right?
[14:15] sustrik keffo: there's one identity per hop, you have to copy all of them to the reply
[14:21] keffo sustrik, do elaborate!
[14:21] keffo before I dice up more code please! :)
[14:21] cremes keffo: i'm writing up a recipe explaining this stuff right now
[14:22] cremes give me another 15 minutes
[14:22] keffo sounds awesome
[14:22] keffo sustrik, I get what you mean, it puzzled me earlier, but I assumed zmq would pass along the identity on its own
[14:23] cremes keffo: it does but *only* for REQ and REP sockets
[14:23] guido_g not with X* sockets
[14:23] cremes XREQ and XREP make you do some of the message routing work yourself
[14:26] sustrik well, it's kind of separated into 2 layers
[14:27] sustrik the bottom layer is XREQ/XREP
[14:27] sustrik what it does is that each hop prepends the message with a new part containing the identity of the sender
[14:27] sustrik that's for requests
[14:27] sustrik for replies it works other way round
[14:28] sustrik at each hop one message part is consumer and used to route the message to the next hop
[14:28] sustrik so, when you have a look at the request after couple of hops
[14:28] sustrik what you see is a backtrace stack of identities
[14:28] sustrik followed by the original message
[14:29] sustrik as for REQ/REP
[14:29] sustrik what they do is very simple
[14:29] keffo indeed, makes sense
[14:29] sustrik REQ prepends the message by an empty message part
[14:30] sustrik so that REP socket on the other side knows where's the bottom of the stack
[14:30] keffo samples would be nice though, like a client server does >1>N hops?
[14:31] sustrik REP socket then chops the backtrace stack off the request and stores it (it knows it'll end by empty message part because it assumes there's a REQ socket on the other side)
[14:31] sustrik when the reply is sent it glues the backtrace stack at the beginning of the reply
[14:31] sustrik that's it
[14:31] sustrik keffo: what samples?
[14:32] keffo A sample that uses two queues for instance, and only using xreq/xrep
[14:32] keffo like, client>queue>worker for instance, all using x's
[14:33] keffo or rather, 2N queues, since that's what I'm doing :)
[14:33] keffo sigh.. N queues, not 2N :)
[14:33] sustrik :)
[14:33] sustrik ok, say 2 queues
[14:34] sustrik so the requester sends message X
[14:34] sustrik on first queue you'll get id1|X
[14:34] sustrik in second queue you'll get id2|id1|X
[14:34] sustrik on the terminal node you'll get id3|id2|id1|X
[14:35] sustrik then you send a reply Y like this: id3|id2|id1|Y
[14:35] keffo yup yup
[14:35] sustrik etc.
[14:35] keffo so each queue would need to keep track of all incoming, and then match with the outgoing?
[14:36] sustrik no
[14:36] sustrik all the state is in message itself
[14:36] sustrik once you forward the message you can forget about it
[14:37] keffo hum, so 2 queuedevices can be linked straight away out of the box?
[14:37] keffo or, N
[14:37] sustrik yes
[14:37] sustrik that's the whole point of the design
[14:37] sustrik it's hop-agnostic
[14:38] mankins hi. anyone around to answer a pub/sub zeromq question?
[14:39] sustrik sure
[14:39] keffo sustrik, Then I'm at a loss again why this stuff fails to deliver?
[14:39] mankins gr8. i'm trying to setup a pub/sub and am a little confused if its possible to bind+subscribe
[14:39] sustrik keffo: how do you create the reply?
[14:40] sustrik mankins: yes
[14:40] cremes keffo: take a look at this draft and let me know what youthink... http://www.zeromq.org/recipe:new-recipe
[14:40] sustrik bind/connect is completely orthogonal to pub/sub
[14:40] mankins sustrik: ok, good, I must have a problem somewhere else then. That's how I read it, but the demo didn't have that.
[14:40] keffo cremes, nice, I'll be back a bit later :)
[14:41] cremes ok
[14:41] mankins i'm also using the perl library which doesn't seem to have a poll function.
[14:43] keffo cremes, more please :)
[14:43] sustrik mankins: what's your problem?
[14:44] cremes keffo: yessir; when i'm done feel free to give specific feedback on questions that are not adequately answered
[14:44] mankins sustrik: no traffic
[14:45] mankins i'm in the middle of debugging, so not quite sure yet...when I wrap my head around it, I'll come back
[14:45] keffo cremes, Will do! Gonna take a shower, then reload the page and see?
[14:45] cremes sure
[14:47] sustrik mankins: have you subscribed?
[14:47] sustrik zmq_setsockopt (s, ZMQ_SUBSCRIBE, "", 0);
[14:47] mankins sustrik: I have two application processes, one which binds to a socket, and sets zmq_subscribe, ""
[14:47] mankins then that process just does sock->recv
[14:48] sustrik what's the socket type?
[14:48] mankins zmq_sub
[14:48] sustrik that's ok
[14:48] sustrik however, are you aware that pub/sub is unreliable?
[14:48] mankins then the other process connects to the other socket (is that the right word for an endpoint?)
[14:49] sustrik i.e. you won't get messages that were sent before you've subscribed?
[14:49] mankins it does a connect, and then send.
[14:49] mankins i'm not getting any messages, either way
[14:50] mankins actually, I think I might have got it to work by sending multiple messages.
[14:51] guido_g how so?
[14:51] mankins yeah, if I wrap the sender-side in a while loop, I can receive the messages, but if the publisher sends, then dies, the message doesn't go thru
[14:51] sustrik i would say the problem is that it takes some time to connect
[14:52] mankins ohhh. is there a way to figure out "readiness"
[14:52] sustrik nope
[14:52] mankins i see. okay...I'll work with that. thanks for your help.
[14:52] sustrik np
[15:11] mankins I'm trying to wrap an eval + alarm combo around the perl library's recv function to timeout, but this doesn't seem to be working...is it possible the program execution is in another thread that doesn't receive the signal?
[15:14] Steve-o There was a discussion about signals before, they should be explicitly ignored so that the Perl thread should catch all of them
[15:15] mankins Steve-o: that sounds like how it should work. I wonder if the current CPAN module does that.
[15:15] Steve-o as in, ZeroMQ already explicitly "ignores" signals, as in sets them to be ignored in libc
[15:16] Steve-o if ZeroMQ is in another thread and the alarm signal is raised for the process it should bounce around till a thread doesn't ignore it
[15:17] Steve-o probably worth trying adding a signal handling inside ZMQ to see if it receives the alarm
[15:18] sustrik Steve-o: the current state of affairs is that 0MQ's internal threads block all the signals
[15:18] sustrik the application threads are untouched
[15:19] sustrik the unclear part is whether 0MQ API should return EINTR in case a signal arrives during a blocking operation
[15:19] sustrik cremes: the next is not entirely right
[15:19] Steve-o mankins: could try siggetmask() to find the state of the Perl thread
[15:19] mankins Steve-o: will look into that.
[15:19] sustrik it's only the XREP socket that adds/removes identities
[15:20] cremes ah, ok
[15:20] cremes will fix
[15:20] sustrik XREQ socket leaves the messages alone
[15:20] sustrik that way you have only a single identity in the stack per intermediary node (such as queue)
[15:20] cremes well, it *adds* its identity to each message
[15:20] cremes XREQ adds, XREP pops, yes?
[15:21] sustrik actually, XREQ does nothing
[15:21] cremes s/adds/pushes/
[15:21] sustrik XREP pushes identity of the peer to the requests
[15:21] sustrik it's a bit strange but it saves bandwidth
[15:22] cremes oh, that isn't clear from observation
[15:22] sustrik ie. in client/server topology, there's no identity passed on the wire
[15:22] cremes so XREP pushes the identity upon receipt
[15:22] sustrik yeah, you would have to inspect the traffic
[15:22] sustrik right
[15:22] cremes so who does the identity pop?
[15:22] sustrik XREP
[15:22] sustrik for replies
[15:23] cremes ok
[15:23] sustrik so it's XREP in both cases:
[15:23] sustrik push on request
[15:23] sustrik pop on reply
[15:23] cremes got it
[15:26] cremes page is fixed; please verify the part that was wrong is now either gone or correct
[15:27] mankins do I need two sockets, one for writing, and one for reading? can I change between pub and sub?
[15:27] cremes mankins: two sockets
[15:27] mankins two contexts too?
[15:27] cremes you can't change a socket type once it is allocated
[15:28] cremes mankins: maybe; i don't know what you have discussed here previously
[15:28] cremes you can have 100s or thousands of sockets per context
[15:28] cremes (actually, i think the limit is 512 right now)
[15:28] cremes per context
[15:33] sustrik cremes: the picutre still shows four identities in the message
[15:33] sustrik whereas there should be 3 as there are 3 XREP sockets
[15:33] cremes ok
[15:34] sustrik also, i would mark the process boundaries on the diagram
[15:34] sustrik that way it would be obvious you have one identity per process
[15:34] sustrik to route back
[15:35] sustrik so, in your case you have a sender process, queue1, queue2 and receiver processes
[15:35] sustrik when the request arrives at receiver
[15:35] sustrik it looks like this:
[15:36] sustrik queue2|queue1|sender|data
[15:36] cremes shouldn't it be: queue2|queue1|sender|null|data
[15:37] sustrik ah, there's a REQ socket?
[15:37] sustrik if so, yes, REQ socket adds empty message part to the bottom of the stack
[15:38] cremes no, but how else do you know when you have moved from routing parts to body parts?
[15:38] cremes it seems like a reasonable default to always use a delimiter
[15:38] sustrik but who will do that?
[15:39] cremes the "source" prepends the null part before sending the body
[15:39] sustrik yes
[15:39] cremes and source is an XREQ
[15:39] sustrik :)
[15:39] cremes heh
[15:39] cremes let me rephrase
[15:39] sustrik then the stack would look like this:
[15:40] sustrik queue2|null|queue1|null|sender|null|data
[15:40] sustrik it's the application that has to add the delimiter
[15:40] sustrik or a REQ socket
[15:40] cremes right, the app adds the delimiter
[15:40] cremes that's how i did it anyway
[15:40] sustrik ok
[15:40] keffo oh, I need to nullterminate each response?
[15:41] cremes keffo: no
[15:41] cremes sustrik thought i meant that each XREQ had to add a null
[15:41] cremes that isn't what i meant
[15:42] cremes only the source XREQ prepends the null; all queues in between just pass the message on
[15:42] cremes the destination XREP uses the null to delimit between the routing parts and the application-level body parts
[15:42] cremes make sense?
[15:43] sustrik cremes: you should start talking in terms of processes
[15:44] keffo yes
[15:44] sustrik if you say "source XREQ prepends the null"
[15:44] sustrik it sounds like 0mq does it
[15:44] keffo agreed
[15:44] cremes i'll fix that
[15:44] keffo and imo, C should be the defacto language for describing things like this?
[15:45] keffo or at the very least, pseudo :)
[15:45] cremes keffo: ruby and python are pretty close to pseudo-code already
[15:45] sustrik C is too messy
[15:45] cremes feel free to edit the page when i'm done and rewrite the code however you like ;)
[15:46] alfborge howfuckedismydatabase.com
[15:46] guido_g oh spam
[15:46] keffo cremes, If I try to rewrite code from a language I dont know, nobody will understand :)
[15:46] alfborge guido_g :)
[15:46] guido_g no joke
[15:47] alfborge sorry then.
[15:47] keffo on a completely different note, the cute little fade-in when you expand the table of contents seems to be buggy in firefox :)
[15:47] AndrewBC alfborge, I find it ironic that that page spat out a "Warning: pg_connect() unable to connect to PostgreSQL server: FATAL 1: IDENT authentication failed for user "postgres" in /var/www/database/postgres/common.php on line 10"
[15:47] AndrewBC Unless that's intentional as a joke
[15:47] alfborge AndrewBC: Actually, that's what makes it really funny.
[15:48] alfborge AndrewBC: Check the other warnings/errors.
[15:48] AndrewBC Okay, yeah. Each one does a topical error.
[15:48] AndrewBC I was too busy debating whether that was intentional to laugh. :|
[15:54] keffo cremes, Why does the page link to itself, under "see also"?
[15:55] cremes i have no idea; i can't edit that part
[15:55] cremes must be part of the page template
[15:59] pieterh keffo: that's just part of the page template
[15:59] pieterh i can fix it
[16:02] keffo ;)
[16:02] pieterh ok, that's fixed, all pages should update in a few seconds
[16:02] keffo I need to sit down and write some idiot-samples for myself to understand what I need to do here
[16:11] mankins it feels like buffers aren't getting flushed for single-message communications. is that possible?
[16:15] mankins and getting segfaults. should I be doing this in C?
[16:20] sustrik mankins: do you have a test program that causes the segfault?
[16:20] mankins i'll work on it. it seems to be random
[16:21] sustrik a sanity check: is your application single-threaded?
[16:21] mankins i just noticed that the perl docs say they are tested with zeromq 2.0.7, and I have .8 installed.
[16:21] mankins it is single threaded. i have a fcgi process and an app server
[16:21] sustrik shouldn't matter, the .8 is a stability release
[16:21] sustrik ok
[16:22] mankins at the moment I'm just trying to get them to echo
[16:23] mankins for some reason the fcgi side seems to get stuck at the recv() an strace shows it doing a recv syscall there.
[16:27] mankins i'm connecting to myself, I wonder if that's the problem.
[16:27] mankins two socks, one binds, one connects
[16:27] sustrik that should be ok
[16:27] sustrik obviously, only if you send first, connect afterwards
[16:27] sustrik recv afterwards*
[16:27] mankins it receives msgs from the client okay, but sends don't go out
[16:28] mankins i init, and setup the sockets, then go into a while loop to receive, is that fitting the model?
[16:28] sustrik sure
[16:29] mankins and topics are | delimited, right?
[16:29] mankins topic|msghere
[16:29] sustrik that's up to you
[16:29] sustrik 0mq does not care
[16:29] mankins ah
[16:29] mankins good to know.
[16:29] mankins then that really shouldn't matter.
[16:30] mankins is there a way to turn on debug to see what happens on send?
[16:30] sustrik on the wire?
[16:30] mankins y
[16:30] sustrik you may use wireshark
[16:30] sustrik to capture the traffic
[16:30] sustrik or tcpdump
[16:30] mankins ok, let me start that up. so no app-level debugging flags?
[16:31] sustrik nope
[16:40] mankins i'd listen on the TCP port, right? the transport isn't udp
[16:40] mankins nvmind
[16:44] sustrik tcp
[16:45] mankins wow, as soon as the communication starts, it doesn't stop. lots of packets.
[16:45] mankins i ran a few seconds and got 2449... but i was just trying to send a 10 byte message and receive it.
[16:46] sustrik are you sure you are not monitoring random traffic
[16:46] sustrik ?
[16:46] Steve-o like monitoring your own SSH connection isn't a good idea
[16:46] sustrik cremes: still here?
[16:47] cremes yes
[16:47] sustrik i've drawn a picture...
[16:47] sustrik wait a sec
[16:48] mankins tcpdump -s 1000 -vvv -X 'tcp port 9010' -i lo
[16:48] sd88g93 greetings
[16:48] sustrik cremes: http://www.zeromq.org/local--files/recipe:new-recipe/x.png
[16:48] sustrik does that help?
[16:49] cremes yes
[16:49] sustrik if needed, you can place it on the wiki
[16:49] sustrik [[=image x.png]]
[16:49] cremes question...
[16:50] sustrik yes?
[16:50] cremes in the queue box you have "req||X" by the down arrow
[16:50] sustrik right
[16:50] cremes where does the "req" identity come from?
[16:51] sustrik XREP socket attaches it to the message
[16:51] cremes ok, so then "X" is the body?
[16:51] sustrik yes
[16:51] cremes ok
[16:51] sustrik i should have named it body
[16:51] sustrik or data
[16:51] cremes that pic looks good
[16:52] sustrik are you on linux?
[16:52] cremes i'll add it along with a little legend so it's easier to interpret
[16:52] cremes osx
[16:52] sustrik ok
[16:52] sd88g93 is it possible to have 2 queeus set up feeding into the same socket ? or is that a bad idea ?
[16:52] sustrik i thought of sending you the source for the picture but it's a random linux drawing tool
[16:53] cremes too bad you didn't put two queues in the picture; that way i could have saved myself from rewriting the explanation
[16:53] sustrik cremes: let me do that
[16:53] cremes great
[16:53] sustrik any other changes?
[16:53] sustrik sd88g93: it's ok
[16:53] cremes make "X" into "body" and "Y" into "ResponseBody" or something similar
[16:53] sustrik the latter is too long
[16:54] cremes sure, pick something better than Y
[16:54] sustrik would stretch out of picutre
[16:54] sd88g93 sustrik: is there any special considerations to use ?
[16:54] cremes Y -> Reply
[16:54] sustrik wouldn't it be confused with the identity "rep"
[16:54] sustrik maybe change the ids to
[16:55] sustrik "app1"
[16:55] sustrik "queue"
[16:55] sustrik "app2"
[16:55] sustrik and bodies to:
[16:55] sustrik "request"
[16:55] sustrik "reply"
[16:56] sd88g93 oh tutorials added into a wiki, neat !
[16:57] cremes sustrik: that's better; any confusion in the picture can be cleared up with a legend for it
[16:58] sustrik ok, working on it
[17:01] mankins something is amiss. if the thing that binds to the socket goes down, shouldn't the tcp connection go down as well? it looks like the client re-binds the port?
[17:01] mankins or is "connect" and "bind" not related to the underlying tcp sockets?
[17:03] cremes mankins: connect & bind are not directly related to the (potentially many) TCP sockets behind the 0mq socket
[17:03] cremes take a look at the guide for a good explanation
[17:03] cremes http://www.zeromq.org/docs:user-guide
[17:03] mankins cremes: thanks, will read.
[17:04] sustrik cremes: http://www.zeromq.org/local--files/recipe:new-recipe/x.png
[17:04] sustrik (reload it to get rid ofcached image)
[17:05] cremes that looks *good*
[17:08] mankins i wonder if I need to do the nanosleep before another read?
[17:08] cremes how do i make the funky 0 with the line through it? what's the key combo?
[17:09] guido_g under linux its built in :)
[17:09] mankins on the mac it's alt o
[17:09] mankins ø
[17:09] cremes mankins: thanks
[17:09] mankins Ø or alt-shift-o
[17:20] Zao U+00D8
[17:56] mankins huh, looks like you can do recv(ZMQ_NOBLOCK) to do non-blocking recvs.
[17:57] sustrik :)
[18:20] mankins in case you're following along at home: I got the REQ/REP type messaging to work, but PUB/SUB would not. I haven't been able to isolate the problem, but am forging forward.
[20:23] andrew_cholakian cremes, if you're around I had a quick question about ffi-rzmq and xrep
[20:24] cremes ask away
[20:24] andrew_cholakian so, ywhen you call say copy_out_string on an xrep socket, it seems like each message comes in three parts
[20:24] andrew_cholakian the identifier, an empty string, and the actual message
[20:24] andrew_cholakian correct?
[20:25] cremes right
[20:26] cremes the identity, null msg and body are all separate ZMQ::Message objects
[20:26] andrew_cholakian oh, interesting
[20:26] cremes every time you call #recv, it returns a ZMQ::Message object
[20:26] cremes have you seen it do something else?
[20:27] andrew_cholakian no, not at all, just curious about it, because the other socket types aren't broken up like that
[20:27] cremes sure they are
[20:27] cremes for example, a SUB socket will return two message parts for each message
[20:27] cremes part 1 is the topic, part2 is the body
[20:28] cremes the bindings aren't doing anything special; they are returning exactly what the 0mq framework is generating
[20:28] cremes ZMQ::Message is really just a wrapper for the zmq_msg_* functions
[20:29] andrew_cholakian gotcha
[20:30] andrew_cholakian so, http://gist.github.com/552184
[20:30] andrew_cholakian so is this wrong here, the way I'm doing things
[20:31] andrew_cholakian since the messages appear in a single part
[20:33] cremes they are coming in as a single message because you are only sending a topic
[20:34] cremes generally you do something like...
[20:34] cremes pub_sock.send_string("Important", ZMQ::SNDMORE)
[20:34] cremes pub_sock.send_string("Find Time Machine")
[20:34] cremes that sends two message parts which constitute a single logical message
[20:34] cremes make sense?
[20:34] andrew_cholakian makes sense
[20:35] andrew_cholakian perfect sense
[20:35] cremes good
[20:35] cremes then on the recv side you can change your handling based upon the received topic
[20:35] cremes the first msg part will contain the topic and subsequent parts contain the body of the msg
[20:35] cremes i hope that helps
[20:35] cremes feel free to add some docs! :)
[20:35] andrew_cholakian that does help actually
[20:36] andrew_cholakian I will, I'm actually working on writing documenting ruby ffi-rzmq
[20:36] cremes then you are my favorite person right now
[20:36] andrew_cholakian I'm doing a presentation to LA ruby on it in a couple weeks
[20:37] andrew_cholakian I want to get a nice collection of minimal examples for every socket class, and some example apps using a mix of em up
[20:37] cremes that sounds great
[20:37] cremes btw, you are kind of cheating in that example you pasted; sockets should be used from the same thread where their context came from
[20:37] cremes that's a 0mq requirement
[20:38] cremes the 2.1.x branch that martin is working on will allow you to move sockets between threads
[20:38] cremes unless i am misunderstanding something
[20:38] cremes perhaps sustrik can chime in
[20:38] andrew_cholakian hmmm, so I should create two contexts then
[20:38] cremes right, one per thread
[20:39] cremes er... one per application thread
[20:40] andrew_cholakian that may explain why running that example requires me to kill -9 it to leave
[20:41] cremes um.... maybe
[20:41] cremes it's probably just MRI's crazy thread handling
[20:41] cremes it works fine with jruby (no hang)
[20:41] andrew_cholakian nah, that was 1.9.2
[20:41] andrew_cholakian MRI, I think it doesn't run period
[20:42] cremes MRI = Matz Ruby Interpreter
[20:43] cremes so MRI is short-hand for 1.8.7, 1.9.1, 1.9.2 etc
[20:43] andrew_cholakian oh, I thought people usually call 1.9.2 YARV
[20:43] cremes yeah, they call it that too :)
[20:43] andrew_cholakian I'm guessing it still has the same threading issues then
[20:43] cremes 1.9.2 has broken thread handling just like 1.8.6, 1.8.7 and 1.9.1
[20:44] cremes it doesn't capture signals properly for external threads, so ctrl-c goes to never never land sometimes
[20:44] cremes and 0mq is the source of the external threads
[20:44] cremes fyi
[20:44] andrew_cholakian makes sense, yeah. I've been using killall -9 ruby quite a bit dealing with 1.9.2
[20:46] cremes i recommend jruby for use with 0mq; it has sane thread and signal handling
[20:46] cremes the main downside is it takes an extra second to start up
[20:49] andrew_cholakian yeah, I've had some issues with jruby and some stuff, I think related to eventmachine when combining them
[21:06] merimus I'm attempting to create a simple multithread pub/sub using python but am getting lots of crashes. Would anyone be kind enough to glance at my code? http://codepad.org/dRRqkeZO
[21:09] merimus starting to wonder if there isn't a bug in pythons threads
[21:13] cremes merimus: take a look at the Thread Safety section on this page: http://api.zeromq.org/zmq.html
[21:13] cremes Each ØMQ socket belonging to a particular context may only be used by the thread that created it using zmq_socket().
[21:14] cremes you should only access that socket from the thread where it was created (the main thread)
[21:17] merimus hmm... then how would I send from multiple threads?
[21:17] cremes each thread needs its own socket
[21:17] cremes let me rephrease
[21:18] cremes each thread needs to create its own socket; that socket should then only be used from that thread
[21:19] merimus but a socket is created on a particular port... which then cannot be reused. Unless I'm missing something
[21:19] cremes no, that's right
[21:19] cremes the way you get around this is to use a zmq_device
[21:19] cremes for the case of PUB/SUB, use a forwarder device
[21:20] cremes the forwarder device binds to two ports, say 5500 and 5501
[21:20] cremes all of your publisher sockets then connect to 5500
[21:20] cremes and all of your subscribers connect to 5501
[21:20] cremes the forwarder device aggregates all of the traffic from every publisher and sends to all subscribers
[21:20] cremes make sense?
[21:21] merimus indeed... heading back to the docs now
[21:21] cremes i highly recommend curling up with the guide for an hour or two: http://www.zeromq.org/docs:user-guide
[21:21] cremes it covers all of the basics and then some
[21:22] merimus had just started on that. Thank you very much.
[21:23] cremes you are welcome

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy