Wednesday April 13, 2011

[Time] NameMessage
[03:39] ilya can someone help by explaining to me which topology I should choose? I'm looking to use 0mq as an remote control agent on the server. Numerous clients will connect and issue commands. Server will perform various commands and reply. pub/sub would be fine if we didn't need the reply piece, but I'd like to be able to use more of req/rep style, but without blocking the server.
[03:41] ilya I can manage threads/processes myself and start up enough processes to handle the requests with req/rep, but I'm wondering if there is a way the 0mq can manage threads/processes. (I'm using python bindings)
[03:48] ilya I've skimmed through the docs, but don't see much comprehensive info/examples. Thanks.
[06:27] benoitc are some people using zeromq for internal message exchange in their appplication ?
[06:28] benoitc i wonder if it could be areplacement to the queue I use to rewrite data data coming from one tcp source and sent to another
[06:33] sustrik yes, if you don't care about using 0MQ's wire protocol on top of TCP
[06:39] benoitc sustrik: well i've a tcp proxy that do transparent A -> B B->A, i would like to introduce a rewrite process between so I can do A -> C -> B C is rewriting dta from A before they are sent to B
[06:41] benoitc I wat to let the possibility to allows C to send data in a streaming fashion, so my idea is to introduce a REQ/REP zmq socket in the middle rather than queing messages and consume the queue in C and aving another queue filled by C that i consime to send to B
[06:41] benoitc not sure I'm clear
[07:11] cyball hi .... i've tried to start an ZMQQueue with jzmq but it seems not to work is there something known about that ? ... i use the ZeroMQ 2.1 GIT - Version and also the JZMQ GIT Version and the platform is OSX 10.6.7 ... i also have tried it on Gentoo the same result there is no binding happen for front and backend XREP and XREQ ... thx
[07:23] sustrik cyball: you should ask on the mailing list
[07:23] sustrik i am not sure any java guys are here
[07:23] sustrik benoitc: not really, sorry
[07:24] benoitc sustrik: basic idea is that i need to rewrite teh reqwest before sending it to the remote
[07:24] benoitc so this rewrite should be on the fly
[07:24] sustrik sure
[07:24] cyball sustrik: thx for the info :-)
[07:24] sustrik no problem with that
[07:25] benoitc and i have this weird idea to use a zmq socket instead of impleting my own app level using 2 queue (one is to put the request and the second to publish the rewrite on the fly before it is sent
[07:25] benoitc but it looks like over engineered :)
[07:28] sustrik it's easier to create a device of your own
[07:28] sustrik the device code is trivial
[07:28] sustrik some 20 lines of code
[07:28] sustrik you can add your rewrite logic to there
[07:33] benoitc mm not sure to follow , you mean putting my own code rather than using zmq ?
[07:48] benoitc i'm lost now :)
[07:50] benoitc sustrik: are you speaking about zmq device here
[07:50] benoitc ?
[07:50] sustrik benoitc: 0mq "device" is a concept
[07:50] sustrik you can either use one delivered with the library
[07:51] sustrik or write your own on top of 0MQ sockets
[07:51] sustrik the implementation of device is trivial
[07:51] sustrik let me find the code...
[07:52] sustrik
[07:52] benoitc k thanks
[07:52] sustrik it's C so it's rather long
[07:52] sustrik but most of it is just error handling and somesuch
[07:52] benoitc sorry my question was more if you were still speaking about zmq, was last a minute or two :)
[07:52] benoitc looking at the code
[07:53] sustrik my point is you can take that code and fill in youtr rewrite logic
[07:53] sustrik if you are not using multipart messages the code gets even simpler
[07:54] benoitc ok looks good, thanks a lot :)
[11:46] NikolaVeber erlang bindings benchmark over the gbit network
[11:52] sustrik NikolaVeber: what about linking the article from
[11:52] NikolaVeber sure, forgot it, sorry
[11:52] sustrik There's a "performance" section there...
[11:53] sustrik
[11:53] locklace is there a proper wire protocol spec for zeromq
[11:54] sustrik over tcp?
[11:54] sustrik have a look at zmq_tcp(7)
[11:54] locklace over everything, ideally
[11:54] sustrik then there's protocol for packet based unreliable transports
[11:55] sustrik can be found at zmq_pgm(7)
[11:55] locklace no udp transport support yet, is that right?
[11:55] sustrik nope
[11:56] sustrik you are welcome to give it a try :)
[11:56] locklace tcp is pretty heavyweight/restricted for a lot of app classes, so ability to run zmq over other internet transports would be very useful
[11:57] locklace but it's not clear to me how much of the reliability logic would then need to be added to zmq itself
[11:57] bwaine hello
[11:57] sustrik what's internet transport?
[11:57] sustrik hi
[11:57] locklace anything running over ip
[11:57] sustrik tcp?
[11:57] locklace sctp is probably a better actual fit for a lot of what zmq is used for, other than that most deployed networks don't know how to deal with it
[11:58] bwaine a quick question. Im using the PHP bindings. Has anyone created a class file netbeans can use to provide auto complete for ZMQ?
[11:58] bwaine If not I'll be happy to make it :0
[11:58] sustrik mpales was contemplating writing an sctp transport
[11:58] sustrik you may ask him what's the status quo
[11:58] locklace it's pretty silly to start (as one usually does) with message-based delivery requirements, and then have to convert messages to/from the tcp "byte stream" abstraction
[11:59] NikolaVeber sustrik, I have linked the results section in the blogpost. This isn't as comprehensive as other benchmarks, I'm not sure if it belongs on that site..
[12:00] sustrik why not?
[12:00] sustrik even if it's not perfect, it's still something people can benefit from
[12:00] NikolaVeber ok :)
[12:01] NikolaVeber can I link it myself?
[12:01] sustrik sure
[12:01] sustrik go on
[12:03] sustrik NikolaVeber: looks like you are from formet yougoslavia; where from exactly?
[12:03] sustrik former*
[12:03] NikolaVeber serbia
[12:03] NikolaVeber cool!
[12:03] NikolaVeber I was in bratislava, loved it!
[12:03] sustrik never been in beograd
[12:03] sustrik one day i'll get there
[12:04] NikolaVeber hehe, I organized java conferences there a couple years ago
[12:04] locklace am i alone in thinking that the current zmq architecture blurs layer/functional boundaries that would better be cleanly separated? specifically, transport-type functions like socket reliability and topology with message routing
[12:04] NikolaVeber we might have had a zmq talk, but I wasn't working with it back then
[12:04] sustrik next time
[12:05] NikolaVeber sure :)
[12:05] sustrik :)
[12:05] NikolaVeber im running an erlang -> c++ test right now, I'll make an update and post all together..
[12:05] sustrik locklace: there's no reliability in 0mq atm
[12:05] locklace iow it looks to me like zmq wants to be both a "network layer" and a "transport layer" but mixes the issues between the two together. maybe i just don't "get it" yet
[12:06] sustrik it's meant to be an L5 layer, above the two mentioned
[12:08] locklace well sure, that's where it sits in practice, but in reality what you have is basically another network+transport layer that sits on top of the existing L3/L4
[12:08] locklace tcp connections (for instance) just become your links at this new layer
[12:08] sustrik true
[12:09] sustrik i don't see an alternative though
[12:09] locklace and you end up having to solve all the same kinds of problems that existing L3/L4 layers do, like addressing/identifiers, routing, and end-to-end message delivery semantics
[12:09] locklace oh, it's not a criticism; it's in a sense the only possibility :)
[12:10] locklace if there's a criticism here, it's only that zmq doesn't appear to explicitly recognise this fact, and split its functionality up accordingly along well-known boundaries
[12:10] sustrik the problem is that 0mq deals with routing based on business logic
[12:10] sustrik whereas L3 deals with routing based on network logic
[12:11] sustrik it's not clear whether the two can be merged in any way
[12:11] locklace i think (in short) that they can and should be
[12:11] sustrik how?
[12:12] locklace there's a class of applications that would really want/need to take advantage of technical network-layer routing functionality; but zmq currently doesn't support that so you'd have to reinvent routing protocols on top of it
[12:12] sustrik the protocols on 0mq layer are based on business logic
[12:12] sustrik that's not visible at L3
[12:13] sustrik if you try to merge the two
[12:13] sustrik you have to either propagate business logic to L3
[12:13] sustrik virtually asking network admins to understand business logic of the apps running on top of the network
[12:14] sustrik or other way round, you would have to implement network routing at L5
[12:14] sustrik which means duplicating the whole IP infrastructure
[12:15] locklace that zmq does bus-log routing/topology is a great strength of it, but in its current form it blurs over all kinds of useful and well-established layer stack distinctions, like network vs. transport functionality
[12:15] sustrik feel free to propose an alternative :)
[12:15] locklace well, what i had in mind was closer to your second option - implementing a proper network layer in zmq - but one that wouldn't just duplicate ip
[12:16] sustrik you'll need thousands of manyears to get anywhere near the level the TCP/IP stack is at now
[12:16] locklace it looks to me (correct me if i'm wrong) that you're starting to run into a set of classic issues related to how you handle addressing/identifiers/routing, which you are having to grapple with, and which arise because you don't have a proper network layer
[12:17] sustrik depends on what you mean by "proper network layer"
[12:17] sustrik 0mq routing is very different from IP routing
[12:17] sustrik as it deals with "business topologies"
[12:18] sustrik such as "NASDAQ stock quote feed"
[12:18] sustrik these topologies can contain large amount of nodes
[12:19] sustrik and the routing within the topology is based on an algorithm called "messaging pattern"
[12:19] sustrik there's no equivalent at L3
[12:20] locklace is it possible to connect a node up to a zmq network and send a message to another node identified by a globally-unique id that gets there efficiently?
[12:20] sustrik nope
[12:20] sustrik it works as a cloud
[12:20] sustrik you connect to the cloud
[12:20] sustrik send a message
[12:20] sustrik message gets somewhere
[12:21] locklace that's one of the fundamental problems/limitations at present imo :)
[12:21] locklace as there is a class of applications where you'd want to rely on the zmq network to do exactly that
[12:21] sustrik why not use standard L3/L4 for those apps?
[12:21] sustrik is seems to fullfil all the requirements nicely
[12:22] locklace i think there are advantages and disadvantages to having a "network layer" that "hard-codes" certain business logic or "messaging" patterns. advantage is it makes it very easy for people who want to just use those common patterns; but it makes things very hard when you want to build new patterns because there isn't a proper layer to bulid them on
[12:22] locklace well, one reason is that the classic socket apis suck :)
[12:23] sustrik ha
[12:23] sustrik i think that's the core of the current discussion
[12:23] sustrik people want to use 0mq as a nice networking framework
[12:23] sustrik a peer to ACE or Boost.Asio
[12:24] sustrik which is a legitimate requirement, but it is not what 0mq is atm
[12:24] sustrik maybe separating the two aspects would be the right solution...
[12:25] locklace i see zmq as potentially solving a very wide class of problems, but in its current form it only solves a subset of that class well (traditional enterprise/business messaging) because that's what it was designed for, and it does it very well. but because it also has a lot of features of a really nice generalised socket architecture/api-replacement, it could do a lot more, and the right way (i think) to do that is to separate its layers
[12:25] locklace more cleanly
[12:25] locklace if that makes sense
[12:25] sustrik yep
[12:25] sustrik generic networking helpers
[12:25] sustrik and the patterns on top of them
[12:26] locklace that would also allow for developer modularity, e.g. so you can have different sets of people working somewhat independently on the different layers and progress them in parallel
[12:26] guido_g utopia here we come
[12:26] locklace basically this is just classic good stack/layering/protocol design
[12:27] sustrik i was never interested in ACE/Boost.Asio side of the thing
[12:27] sustrik but it seems it would be helpful to have such layer
[12:27] sustrik you may try to separate the two
[12:27] sustrik but it's not a trivial task
[12:28] locklace well, since zmq already goes halfway or more toward giving the world a massively improved socket layer, it would be nice to see it evolve fully into that role...
[12:28] sustrik hi guido :)
[12:28] sustrik sure, give it a try
[12:28] guido_g hi sustrik
[12:29] sustrik locklace: one problem you'll encounter is that the 0mq API niceness
[12:29] sustrik is a product of messaging patterns
[12:29] guido_g the world is full of projects with a broad and _general_ scope... unsurprisingly most of them are dead
[12:29] guido_g one of the best points of ømq is it's focus
[12:29] sustrik by selecting a pattern, 0mq can do a lot of logic for you and never bothers you about it
[12:30] sustrik when you start exposing all the details you need for generic messaging via API you end up with something very similar to BSD socket API
[12:30] sustrik guido_g: well, that's my focus
[12:30] locklace so you'd end up with a complete and well-defined socket layer and api (effectively a new network+transport layer that treats classic L4 circuits as its links, and allows you to request different kinds of delivery semantics), and then a layer of "messaging patterns" on top of that which simply crystallise the most common use cases, but which can then easily be extended
[12:31] sustrik people are free to to try their own ideas
[12:31] sustrik but somehow that doesn't happen :)
[12:31] guido_g what a surprise
[12:31] sustrik :)
[12:32] sustrik locklace: try to do that; but recall the main motivation was that BSD socket API sucks
[12:32] sustrik so if you end up with equivalent or worse API, you've failed to achieve the goal
[12:33] locklace you could be right. it's certainly plausible that attempting to generalise zmq would be a regression and not an improvement. there's no question it would be tricky, but since it's already gone 3/4 of the way there or so, it doesn't look impossible to me
[12:34] locklace i guess what i'd like to know is what the plans are for the future in this regard
[12:34] sustrik no plans myself, but watch for discussion on the mailing list
[12:34] sustrik it seems to be a hot topic :0
[12:34] sustrik :)
[12:35] locklace yeah
[12:35] locklace when i see people grappling with classic network-layer problems like how to handle addressing and routing, then this where my thinking leads :)
[12:37] sustrik well, i'm coming from an enterprise-messaging sphere
[12:37] sustrik so i have enough experience with duplicit functionality
[12:38] locklace heh
[12:38] sustrik 0mq actually evolved from classic enterprise messaging ideas
[12:38] sustrik by removing the duplicities
[12:38] sustrik almost everything didn't passed the duplicty test
[12:38] sustrik and was removed
[12:38] locklace i'm coming from core networking
[12:38] sustrik i've figured out
[12:38] sustrik what remained was messaging patterns
[12:39] sustrik as those cannot be delegated to upper layer (like transactions or serialisation)
[12:39] sustrik or to lower layer, like L3/L4
[12:40] locklace i think that's a good insight
[12:40] sustrik locklace: what's exactly is you area of experience?
[12:40] sustrik i'm kind of missing lower layer people in 0mq community
[12:41] locklace it just looks to me like zmq blurs that layer with what's effectively a "higher" L3/L4 set of functionality that's required to handle routing and reliability semantics. and what i see from present discussions is that it's exactly this area that's giving you trouble at the moment, and it's happening because the layers are blurred
[12:42] locklace sustrik: core network architecture and platform/protocol design
[12:42] sustrik are you involved in ietf by chance?
[12:42] locklace yes
[12:42] sustrik nice
[12:43] sustrik been in prague?
[12:43] locklace very recently :)
[12:43] sustrik heh
[12:43] sustrik i've posted a notice on the mailing list about me being in prague...
[12:43] sustrik we could have had a beer
[12:44] locklace yeah, i didn't actually know about zmq until about two days ago
[12:44] sustrik pity
[12:44] sustrik anyway, my idea is to get patterns standardised in the future
[12:44] sustrik but there's a long way to go still
[12:45] locklace standardised how?
[12:45] sustrik hard to say
[12:45] sustrik one option is
[12:45] sustrik to start with fixing the semantics
[12:46] sustrik and providing that as a means for bridging existing messaging implementations into larger topologies
[12:46] sustrik the other option is to actually focus on the actual wire level protocol
[12:47] sustrik but that's kind of contentious, given there's a lot of existing messaging protocols out there
[12:48] locklace yes, it will be
[12:48] sustrik anyway, i think there'll be a lot of support for that kind of thing in the future
[12:49] sustrik the need for hardware-software interaction can even lead to need for actual binary protocol
[12:49] locklace but i'd still suggest it's the right way to do things; it actually doesn't matter whether X group of people think it's contentious, what matters is the discipline of writing it up in that form and subjecting it to the full community review process that enables
[12:50] sustrik yep
[12:50] sustrik what's your email btw?
[12:50] locklace i.e. if you can write the whole protocol up in draft form so it can be properly reviewed, you'll be able to emerge with a much better and more solid end result
[12:50] sustrik i'll ping you if anything interesting happens in this area
[12:51] locklace ok. i'll send you a note offline
[12:51] sustrik thanks
[12:51] sustrik souls*
[12:52] locklace yeah, but there's a lot of work needed to even get near that point
[12:52] sustrik ack
[12:52] locklace the write-up above and an initial public review cycle, mainly ;)
[12:53] sustrik yep, well get there
[12:54] locklace if you (or someone with the knowledge of zmq) is willing to put the work in to write it all up in draft form, i might be able to help as editor and with some process stuff
[12:54] mikko sustrik:
[12:54] locklace i think you'll also find that once you have it down in that format, it will greatly improve the traction you get on design issues on the mailing list etc.
[12:55] sustrik mikko: i've seen that
[12:55] locklace traction is always *much* easier to achieve when people can comment on specific parts of draft text
[12:55] sustrik there doesn't seem to be an answer
[12:55] sustrik other than "coming soon"
[12:56] sustrik anyway, let's see, maybe the guy will really implement it
[12:56] sustrik locklace: yes, you are right
[12:57] sustrik the biggest problem with an initial draft is who to focus it on
[12:57] sustrik enterprise messaging devs are not participating on ietf
[12:57] sustrik while ietf folks have no idea of enterprise messaging
[12:57] sustrik and, additionally, 0mq-style messaging is no longer an enterprise messaging proper
[12:58] locklace well, my suggestion is to forget about targeting it at any specific group at the moment, but write it up in the format of an ietf draft
[12:58] locklace regardless of where it goes from there, that will be a major step in the right direction even just internally for the zmq project and mailing list
[12:58] sustrik yeah, it should probably be worded in the most generic way
[12:59] sustrik so that anyone can understand it
[12:59] sustrik but then, it would probably look like a vapourware
[12:59] sustrik oh my
[12:59] locklace yes, and writing it up in that format (a) helps you structure it as a proper protocol architecture, and (b) helps focus discussion
[13:01] sustrik ack
[13:02] locklace and of course, if it then does get edited into a shape fit for possible ietf work, that enables you to think about stuff like having a bof with a nonzero chance of success
[13:03] locklace this is probably obvious, but i'll say it anyway: the first thing the draft should address is the set of requirements. in fact, the first version should probably be nothing *but* a set of requirements, which people can then argue over and loosely agree on before anything else is done
[13:04] sustrik yes, that's the plan
[13:04] sustrik the another interesting question is which area it should be dealt with in
[13:05] sustrik so far i am in contact with apps area guys
[13:05] locklace area?
[13:05] sustrik ietf area
[13:05] sustrik but the think is a bit lower layer
[13:05] sustrik somewhere between transport and apps
[13:05] locklace well let's just say there are good reasons why i said not to worry about targeting any particular group at the outset ;)
[13:06] sustrik :)
[13:06] locklace i really would not worry about that kind of thing at present, you have enough work to do without worrying about the organisational politics
[13:06] locklace get a sound technical draft put together first, worry about where it goes to later
[13:07] locklace it will be pretty much impossible to answer the "where should it go" question until the requirements and architecture are clear anyway
[13:09] locklace xml2rfc makes writing things up easy now, yay
[13:09] sustrik yep, i want to learn to use it
[13:09] locklace it will take you five minutes
[13:09] sustrik formatting drafts by hand is a pain
[13:10] locklace just go to, submit the xml and it spits out text. the actual xml schema is almost trivial, hit the "unofficial successor" link at the top of the page to get to the reference, and start with some existing draft .xml file as a template
[13:11] sustrik thx
[13:44] pantsman hi - it's great that there are windows binaries for pyzmq on its github downloads page, but I still need visual studio installed to run them - does this mean they're debug builds?
[13:45] sustrik why do you need msvc?
[13:47] pantsman sustrik, because without, the libzmq.dll that comes with the pyzmq installer can't find a DLL called Microsoft.VC90.DebugCRT
[13:48] sustrik yes, the library was compiled with debug CRT
[13:49] sustrik where have you got the binary from?
[13:49] pantsman I was using
[13:52] sustrik no idea where that comes from
[13:52] sustrik mikko: are you there?
[13:54] sustrik hm, looks like he's not here
[13:54] sustrik pantsman: can you please complain about it on the mailing list?
[13:55] pantsman sustrik, sure
[13:55] sustrik thanks
[13:56] guido_g mentioning pyzmq in the subject, as it seems to be a problem with pyzmq build
[13:58] mikko sustrik: yes
[13:58] NikolaVeber sustrik, I have linked the post at,
[14:04] sustrik mikko: any idea where that binary comes from?
[14:05] sustrik NikolaVeber: thanks
[14:05] NikolaVeber np! :)
[14:08] evax NikolaVeber: the last test is fine, notice the 1e7 on top of the graph :)
[14:08] NikolaVeber oh, yeah! :)
[14:09] NikolaVeber I'll update the note
[14:10] NikolaVeber done
[14:10] sustrik btw, the order of graphs is somewhat chaotic
[14:14] NikolaVeber just done that
[14:18] mikko sustrik: the .msi?
[14:18] sustrik yes
[14:19] mikko must be minrk / bgranger
[14:19] sustrik i see
[14:19] mikko i would imagine
[14:19] mikko Microsoft.VC90.DebugCRT
[14:19] mikko sounds like it's a debug build
[14:19] mikko rather than a release build
[14:19] sustrik or a bad CRT option passed to the compiler
[14:20] sustrik anyway, it's reported on the ML
[14:20] sustrik it's up to pyzmq guys noqw
[14:32] pantsman fortunately not a big problem for me since I can build libzmq.dll myself; however building pyzmq is something that's beyond me at the moment
[14:48] yawniek if i add zmq.SNDMORE with pyzmq the receiver does not get the message imediately but only after the last message of the multipart message has been sent off. shouldnt it be possible to stream multipart messages?
[14:49] yawniek err mybad
[14:57] sustrik yawniek: it's transactional
[14:57] sustrik message is treated as an atomic unit
[14:58] sustrik you either get whole message, or none of it
[14:59] yawniek ok. i want to build an interface for a database where the results should be streamable. so query gets sent and a stream of messages should come back. how should the architecture look like?
[15:00] sustrik the reply doesn't fit into memory, right?
[15:01] yawniek exactely
[15:01] sustrik 0mq holds messages in memory
[15:02] sustrik so it's hard to work with a message that does not fit into it
[15:02] sustrik you can use XREQ/XREP to get a streaming like behaviour
[15:02] sustrik not nice, but doable
[15:02] sustrik check the guide
[15:04] yawniek yeah, im reading it over and over again. i think a two step process with PUSH/PULL socket for the streaming is a way but that gives some overhead
[17:51] yziquel is there any quick and fast way of knowing the ip of the host of a client socket that connects on a server? from the server?
[17:56] erickt yziquel: I don't believe that's exposed from the api
[17:56] yziquel ok. thanks.
[17:58] erickt you can get the identity for certain socket types. That's talked about here:
[18:04] yziquel what i really need is a directory of endpoints which is info to be served by a central server.
[18:04] yziquel erickt: so i'm wondering how to populate it precisely.
[18:05] mikko yziquel: there is a project aiming to do that
[18:06] mikko yziquel: but i think it's only a wiki-page has been written about it this far
[19:01] mpales hi
[19:01] mpales i have a question related to application design
[19:02] mpales anyone here?
[19:03] isvara I am here, but that probably doesn't help you.
[19:09] sustrik mpales: hi
[19:10] mpales sustrik: i am trying to create a messaging pattern and need a help
[19:11] mpales the system should reassemble segmented messages and process them
[19:11] mpales i'd like to have several workers for reassembling
[19:12] mpales the point is that is a worker gets a segment, it should also receive all thge other segments
[19:12] mpales so i cannot use a plain round robin or lru
[19:12] sustrik why not multipart messages?
[19:13] mpales the application has no control over segments
[19:13] mpales they are the input
[19:13] mpales other clients sent them
[19:14] sustrik why don't they sent them as multipart msgs?
[19:14] mpales it's not a zmq based system
[19:15] sustrik ok, what's the problem with 0mq then?
[19:15] mpales how to route those segments
[19:15] sustrik i mean
[19:15] sustrik you get a list of segments
[19:15] mpales all segments with same id should go to the same worker
[19:15] sustrik then you send it to a worker as a multipart message
[19:17] mpales but all segments belong to the same message
[19:17] sustrik ?
[19:17] sustrik i think i don't get it
[19:17] mpales oops, my mistake
[19:18] mpales segments belong to different messages
[19:18] guido_g create a map: source id -> ømq worker
[19:18] sustrik aha
[19:18] sustrik you want stateful services
[19:19] mpales guido: that's also my idea
[19:19] mpales sustrik: yes
[19:19] sustrik there's no easy way to do that in 0mq
[19:19] sustrik the issue is being hotly discussed
[19:19] guido_g mpales: look at the lru examples in the guide
[19:20] sustrik yep, have a look into the guide
[19:20] guido_g they will show you how to deal w/ socket identities
[19:20] sustrik there are various workarounds there
[19:20] mpales sustrik: i actually do not really expect that zmq will support it, but want to be sure :)
[19:20] guido_g and instead of the lru use the source id for routing
[19:20] mpales i already read the whole guide
[19:21] guido_g so?
[19:21] mpales sustrik: since the routing decision is on application level and zmq sits below it
[19:21] sustrik what pieter does in the guide is using XREP socket as a way to route message to specific peers
[19:22] mpales guido_g: yes, that could do it
[19:22] sustrik that kind of helps with stateful services
[19:22] guido_g mpales: it will because your problem is only a variation of the lru example
[19:22] sustrik it basically shifts all the responsibility for routing to the user
[19:23] mpales ok, thanks guys
[19:24] mpales sustrik: the only problem with using xrep i have that it discards messages if they cannot be delivered
[19:24] mpales in my case i woulk prefer blocking
[19:24] mpales if the worker is busy
[19:25] mpales since i _need_ to deliver to that particular worker and prefer to wait over dropping the message
[19:25] sustrik mpales: yes, that's because xrep is meant to be a stateless service
[19:25] sustrik ie. it sends the reply
[19:25] sustrik if the requester is not there anymore it drops the message
[19:26] mpales sustrik: yes, that's clear
[19:26] mpales what i need is a xrep alternative that will block
[19:26] sustrik so, i would say, you have to modify 0mq itself
[19:26] sustrik introduce the blocking behaviour
[19:27] mpales sustrik: i was thinking about a socket type where you can configure its behaviour - to avoid multiple types
[19:28] sustrik if you believe you can deal with all the possible combinations, why not
[19:28] guido_g sustrik: would zmq_send return != 0 in such a case?
[19:29] guido_g i mean when sending to a xrep socket w/ hwm reached
[19:29] sustrik with ZMQ_NOBLOCK flag it returns -1 and EAGAIN
[19:29] sustrik without it, it just blocks
[19:29] guido_g ha! :)
[19:30] guido_g no, i was talking of xrep, where it drops the message
[19:30] sustrik no, it's silent
[19:31] guido_g what if this would be changed?
[19:31] sustrik mpales: would that help?
[19:31] sustrik sounds easy to implement
[19:31] guido_g it still drops the message but the return code signales that the send was *not* done
[19:31] mpales the configuration of that 'generic' socket could be limited to incoming/outgoing routing strategy and hwm option action
[19:31] guido_g like with xreq in NOBLOCK mode
[19:33] mpales sustrik: you're suggesting to set ZMQ_NOBLOCK to the socket?
[19:34] sustrik nope, i was just answering guido_g
[19:34] guido_g mpales: the idea was to make the send on an XREP socket retunr != 0 when the message has been droppen, so that the application has a way to see that the message was not sent
[19:35] mpales guido_g: aha, that would help
[19:36] sustrik mpales: check zmq::xrep_t::xsend
[19:36] sustrik look for this line: if (it != outpipes.end ()) {
[19:37] mpales sustrik: got it
[19:38] guido_g sustrik: any reasons that this should not be done for all socket types that drop silently on send?
[19:39] sustrik this is just a workaround for mpales
[19:39] sustrik in reality it doesn't really woek
[19:39] sustrik work
[19:39] guido_g why?
[19:39] sustrik if you insert a device between sender and receiver, the message may be dropped without sender ever noticing it
[19:40] guido_g i see
[19:41] sustrik it's a "scalability check" principle
[19:41] sustrik should be documented somewhere even
[19:42] guido_g need heavy re-wiring of some brain areas
[19:42] sustrik :)
[19:43] guido_g i'm playing w/ ømq for months now and still "don't get it"
[19:45] sustrik i tend to play "what if it was TCP/IP?" game
[19:45] sustrik if you add a router into middle, would it still work?
[19:46] guido_g yes
[19:46] guido_g it's just that i need to develop this as a reflex
[19:47] guido_g it'S a matter of training i guess
[19:47] sustrik yep, i'm training it for couple of years now
[19:47] sustrik still i sometimes get trapped
[19:47] guido_g *sigh*
[19:49] mpales sustrik: in that code i also need to change the logic to be able to do a resend, right?
[19:50] sustrik well, you should not change the state of socket in any way if you are dropping the message
[19:50] sustrik the question is what to do with subdequent message parts
[19:51] sustrik in case of multipart messages
[19:51] sustrik ie. first one is dropped
[19:51] sustrik do we expect other parts yet to be sent
[19:51] sustrik or do we expect the user to back down and don't send subsequent parts?
[19:52] mpales sustrik: i would suggest the latter
[19:52] sustrik it's up to you
[19:53] sustrik do it in such a way as to make you app benefit from it
[19:56] mpales sustrik: is it possible that only first part is sent and the rest is dropped in case the queue is full?
[19:56] sustrik no
[19:56] sustrik it's atomic
[19:56] sustrik it either sends the whole message
[19:56] sustrik or nothing
[19:57] mpales ok
[20:01] mpales sustrik: i also noticed that ypipe is affected by false sharing in case you have several io threads or several contexts
[20:02] mpales putting cacheline size pads between *w and *r would help
[20:02] sustrik nice spot
[20:02] sustrik would you provide a patch?
[20:03] mpales i can, sure
[20:03] sustrik :)
[20:03] sustrik great
[20:03] mpales but later this week, too busy these days
[20:43] private_meta When using IPC sockets, how do I specify absolut locations of IPC files?
[20:43] private_meta +e
[20:48] sustrik ipc:///my/path
[20:49] private_meta yeah, found it as well, thanks
[21:27] eyeris I've installed zeromq 2.1.4 under cygwin but gcc complains about undefined references when I compile a simple test program (from the guide).
[21:28] eyeris all of the undefined references it complains about begin with _zmq
[21:30] Seta00 are you linking your project with zmq properly?
[21:30] eyeris Yeah, making a paste now
[21:31] eyeris
[21:34] eyeris If I run make check in the zeromq build tree a bunch of them fail and then it seems to hang after test_hwm.exe passes
[23:27] stevan is this a good place to ask about jzmq build issues on windows?
[23:28] stevan or is there a more jzmq specific room?
[23:28] stevan this is what I am getting ->
[23:28] stevan libzmq compiled just fine
[23:29] stevan this is with Visual Studio 2010 too, if that makes any difference
[23:52] stevan ah ha, I got it
[23:53] stevan libzmq HEAD was not what I really wanted, I wanted the one on the downloads page