| [Time] Name | Message |
|
[03:39] ilya
|
can someone help by explaining to me which topology I should choose? I'm looking to use 0mq as an remote control agent on the server. Numerous clients will connect and issue commands. Server will perform various commands and reply. pub/sub would be fine if we didn't need the reply piece, but I'd like to be able to use more of req/rep style, but without blocking the server.
|
|
[03:41] ilya
|
I can manage threads/processes myself and start up enough processes to handle the requests with req/rep, but I'm wondering if there is a way the 0mq can manage threads/processes. (I'm using python bindings)
|
|
[03:48] ilya
|
I've skimmed through the docs, but don't see much comprehensive info/examples. Thanks.
|
|
[06:27] benoitc
|
are some people using zeromq for internal message exchange in their appplication ?
|
|
[06:28] benoitc
|
i wonder if it could be areplacement to the queue I use to rewrite data data coming from one tcp source and sent to another
|
|
[06:33] sustrik
|
yes, if you don't care about using 0MQ's wire protocol on top of TCP
|
|
[06:39] benoitc
|
sustrik: well i've a tcp proxy that do transparent A -> B B->A, i would like to introduce a rewrite process between so I can do A -> C -> B C is rewriting dta from A before they are sent to B
|
|
[06:41] benoitc
|
I wat to let the possibility to allows C to send data in a streaming fashion, so my idea is to introduce a REQ/REP zmq socket in the middle rather than queing messages and consume the queue in C and aving another queue filled by C that i consime to send to B
|
|
[06:41] benoitc
|
not sure I'm clear
|
|
[07:11] cyball
|
hi .... i've tried to start an ZMQQueue with jzmq but it seems not to work is there something known about that ? ... i use the ZeroMQ 2.1 GIT - Version and also the JZMQ GIT Version and the platform is OSX 10.6.7 ... i also have tried it on Gentoo the same result there is no binding happen for front and backend XREP and XREQ ... thx
|
|
[07:23] sustrik
|
cyball: you should ask on the mailing list
|
|
[07:23] sustrik
|
i am not sure any java guys are here
|
|
[07:23] sustrik
|
benoitc: not really, sorry
|
|
[07:24] benoitc
|
sustrik: basic idea is that i need to rewrite teh reqwest before sending it to the remote
|
|
[07:24] benoitc
|
so this rewrite should be on the fly
|
|
[07:24] sustrik
|
sure
|
|
[07:24] cyball
|
sustrik: thx for the info :-)
|
|
[07:24] sustrik
|
no problem with that
|
|
[07:25] benoitc
|
and i have this weird idea to use a zmq socket instead of impleting my own app level using 2 queue (one is to put the request and the second to publish the rewrite on the fly before it is sent
|
|
[07:25] benoitc
|
but it looks like over engineered :)
|
|
[07:28] sustrik
|
it's easier to create a device of your own
|
|
[07:28] sustrik
|
the device code is trivial
|
|
[07:28] sustrik
|
some 20 lines of code
|
|
[07:28] sustrik
|
you can add your rewrite logic to there
|
|
[07:33] benoitc
|
mm not sure to follow , you mean putting my own code rather than using zmq ?
|
|
[07:48] benoitc
|
i'm lost now :)
|
|
[07:50] benoitc
|
sustrik: are you speaking about zmq device here
|
|
[07:50] benoitc
|
?
|
|
[07:50] sustrik
|
benoitc: 0mq "device" is a concept
|
|
[07:50] sustrik
|
you can either use one delivered with the library
|
|
[07:51] sustrik
|
or write your own on top of 0MQ sockets
|
|
[07:51] sustrik
|
the implementation of device is trivial
|
|
[07:51] sustrik
|
let me find the code...
|
|
[07:52] sustrik
|
https://github.com/zeromq/zeromq2-1/blob/master/src/device.cpp
|
|
[07:52] benoitc
|
k thanks
|
|
[07:52] sustrik
|
it's C so it's rather long
|
|
[07:52] sustrik
|
but most of it is just error handling and somesuch
|
|
[07:52] benoitc
|
sorry my question was more if you were still speaking about zmq, was last a minute or two :)
|
|
[07:52] benoitc
|
looking at the code
|
|
[07:53] sustrik
|
my point is you can take that code and fill in youtr rewrite logic
|
|
[07:53] sustrik
|
if you are not using multipart messages the code gets even simpler
|
|
[07:54] benoitc
|
ok looks good, thanks a lot :)
|
|
[11:46] NikolaVeber
|
erlang bindings benchmark over the gbit network http://nikolaveber.blogspot.com/
|
|
[11:52] sustrik
|
NikolaVeber: what about linking the article from zeromq.org?
|
|
[11:52] NikolaVeber
|
sure, forgot it, sorry
|
|
[11:52] sustrik
|
There's a "performance" section there...
|
|
[11:53] sustrik
|
http://www.zeromq.org/area:results
|
|
[11:53] locklace
|
is there a proper wire protocol spec for zeromq
|
|
[11:54] sustrik
|
over tcp?
|
|
[11:54] sustrik
|
have a look at zmq_tcp(7)
|
|
[11:54] locklace
|
over everything, ideally
|
|
[11:54] sustrik
|
then there's protocol for packet based unreliable transports
|
|
[11:55] sustrik
|
can be found at zmq_pgm(7)
|
|
[11:55] locklace
|
no udp transport support yet, is that right?
|
|
[11:55] sustrik
|
nope
|
|
[11:56] sustrik
|
you are welcome to give it a try :)
|
|
[11:56] locklace
|
tcp is pretty heavyweight/restricted for a lot of app classes, so ability to run zmq over other internet transports would be very useful
|
|
[11:57] locklace
|
but it's not clear to me how much of the reliability logic would then need to be added to zmq itself
|
|
[11:57] bwaine
|
hello
|
|
[11:57] sustrik
|
what's internet transport?
|
|
[11:57] sustrik
|
hi
|
|
[11:57] locklace
|
anything running over ip
|
|
[11:57] sustrik
|
tcp?
|
|
[11:57] locklace
|
sctp is probably a better actual fit for a lot of what zmq is used for, other than that most deployed networks don't know how to deal with it
|
|
[11:58] bwaine
|
a quick question. Im using the PHP bindings. Has anyone created a class file netbeans can use to provide auto complete for ZMQ?
|
|
[11:58] bwaine
|
If not I'll be happy to make it :0
|
|
[11:58] sustrik
|
mpales was contemplating writing an sctp transport
|
|
[11:58] sustrik
|
you may ask him what's the status quo
|
|
[11:58] locklace
|
it's pretty silly to start (as one usually does) with message-based delivery requirements, and then have to convert messages to/from the tcp "byte stream" abstraction
|
|
[11:59] NikolaVeber
|
sustrik, I have linked the results section in the blogpost. This isn't as comprehensive as other benchmarks, I'm not sure if it belongs on that site..
|
|
[12:00] sustrik
|
why not?
|
|
[12:00] sustrik
|
even if it's not perfect, it's still something people can benefit from
|
|
[12:00] NikolaVeber
|
ok :)
|
|
[12:01] NikolaVeber
|
can I link it myself?
|
|
[12:01] sustrik
|
sure
|
|
[12:01] sustrik
|
go on
|
|
[12:03] sustrik
|
NikolaVeber: looks like you are from formet yougoslavia; where from exactly?
|
|
[12:03] sustrik
|
former*
|
|
[12:03] NikolaVeber
|
serbia
|
|
[12:03] NikolaVeber
|
cool!
|
|
[12:03] NikolaVeber
|
I was in bratislava, loved it!
|
|
[12:03] sustrik
|
never been in beograd
|
|
[12:03] sustrik
|
one day i'll get there
|
|
[12:04] NikolaVeber
|
hehe, I organized java conferences there a couple years ago
|
|
[12:04] locklace
|
am i alone in thinking that the current zmq architecture blurs layer/functional boundaries that would better be cleanly separated? specifically, transport-type functions like socket reliability and topology with message routing
|
|
[12:04] NikolaVeber
|
we might have had a zmq talk, but I wasn't working with it back then
|
|
[12:04] sustrik
|
next time
|
|
[12:05] NikolaVeber
|
sure :)
|
|
[12:05] sustrik
|
:)
|
|
[12:05] NikolaVeber
|
im running an erlang -> c++ test right now, I'll make an update and post all together..
|
|
[12:05] sustrik
|
locklace: there's no reliability in 0mq atm
|
|
[12:05] locklace
|
iow it looks to me like zmq wants to be both a "network layer" and a "transport layer" but mixes the issues between the two together. maybe i just don't "get it" yet
|
|
[12:06] sustrik
|
it's meant to be an L5 layer, above the two mentioned
|
|
[12:08] locklace
|
well sure, that's where it sits in practice, but in reality what you have is basically another network+transport layer that sits on top of the existing L3/L4
|
|
[12:08] locklace
|
tcp connections (for instance) just become your links at this new layer
|
|
[12:08] sustrik
|
true
|
|
[12:09] sustrik
|
i don't see an alternative though
|
|
[12:09] locklace
|
and you end up having to solve all the same kinds of problems that existing L3/L4 layers do, like addressing/identifiers, routing, and end-to-end message delivery semantics
|
|
[12:09] locklace
|
oh, it's not a criticism; it's in a sense the only possibility :)
|
|
[12:10] locklace
|
if there's a criticism here, it's only that zmq doesn't appear to explicitly recognise this fact, and split its functionality up accordingly along well-known boundaries
|
|
[12:10] sustrik
|
the problem is that 0mq deals with routing based on business logic
|
|
[12:10] sustrik
|
whereas L3 deals with routing based on network logic
|
|
[12:11] sustrik
|
it's not clear whether the two can be merged in any way
|
|
[12:11] locklace
|
i think (in short) that they can and should be
|
|
[12:11] sustrik
|
how?
|
|
[12:12] locklace
|
there's a class of applications that would really want/need to take advantage of technical network-layer routing functionality; but zmq currently doesn't support that so you'd have to reinvent routing protocols on top of it
|
|
[12:12] sustrik
|
the protocols on 0mq layer are based on business logic
|
|
[12:12] sustrik
|
that's not visible at L3
|
|
[12:13] sustrik
|
if you try to merge the two
|
|
[12:13] sustrik
|
you have to either propagate business logic to L3
|
|
[12:13] sustrik
|
virtually asking network admins to understand business logic of the apps running on top of the network
|
|
[12:14] sustrik
|
or other way round, you would have to implement network routing at L5
|
|
[12:14] sustrik
|
which means duplicating the whole IP infrastructure
|
|
[12:15] locklace
|
that zmq does bus-log routing/topology is a great strength of it, but in its current form it blurs over all kinds of useful and well-established layer stack distinctions, like network vs. transport functionality
|
|
[12:15] sustrik
|
feel free to propose an alternative :)
|
|
[12:15] locklace
|
well, what i had in mind was closer to your second option - implementing a proper network layer in zmq - but one that wouldn't just duplicate ip
|
|
[12:16] sustrik
|
you'll need thousands of manyears to get anywhere near the level the TCP/IP stack is at now
|
|
[12:16] locklace
|
it looks to me (correct me if i'm wrong) that you're starting to run into a set of classic issues related to how you handle addressing/identifiers/routing, which you are having to grapple with, and which arise because you don't have a proper network layer
|
|
[12:17] sustrik
|
depends on what you mean by "proper network layer"
|
|
[12:17] sustrik
|
0mq routing is very different from IP routing
|
|
[12:17] sustrik
|
as it deals with "business topologies"
|
|
[12:18] sustrik
|
such as "NASDAQ stock quote feed"
|
|
[12:18] sustrik
|
these topologies can contain large amount of nodes
|
|
[12:19] sustrik
|
and the routing within the topology is based on an algorithm called "messaging pattern"
|
|
[12:19] sustrik
|
there's no equivalent at L3
|
|
[12:20] locklace
|
is it possible to connect a node up to a zmq network and send a message to another node identified by a globally-unique id that gets there efficiently?
|
|
[12:20] sustrik
|
nope
|
|
[12:20] sustrik
|
it works as a cloud
|
|
[12:20] sustrik
|
you connect to the cloud
|
|
[12:20] sustrik
|
send a message
|
|
[12:20] sustrik
|
message gets somewhere
|
|
[12:21] locklace
|
that's one of the fundamental problems/limitations at present imo :)
|
|
[12:21] locklace
|
as there is a class of applications where you'd want to rely on the zmq network to do exactly that
|
|
[12:21] sustrik
|
why not use standard L3/L4 for those apps?
|
|
[12:21] sustrik
|
is seems to fullfil all the requirements nicely
|
|
[12:22] locklace
|
i think there are advantages and disadvantages to having a "network layer" that "hard-codes" certain business logic or "messaging" patterns. advantage is it makes it very easy for people who want to just use those common patterns; but it makes things very hard when you want to build new patterns because there isn't a proper layer to bulid them on
|
|
[12:22] locklace
|
well, one reason is that the classic socket apis suck :)
|
|
[12:23] sustrik
|
ha
|
|
[12:23] sustrik
|
i think that's the core of the current discussion
|
|
[12:23] sustrik
|
people want to use 0mq as a nice networking framework
|
|
[12:23] sustrik
|
a peer to ACE or Boost.Asio
|
|
[12:24] sustrik
|
which is a legitimate requirement, but it is not what 0mq is atm
|
|
[12:24] sustrik
|
maybe separating the two aspects would be the right solution...
|
|
[12:25] locklace
|
i see zmq as potentially solving a very wide class of problems, but in its current form it only solves a subset of that class well (traditional enterprise/business messaging) because that's what it was designed for, and it does it very well. but because it also has a lot of features of a really nice generalised socket architecture/api-replacement, it could do a lot more, and the right way (i think) to do that is to separate its layers
|
|
[12:25] locklace
|
more cleanly
|
|
[12:25] locklace
|
if that makes sense
|
|
[12:25] sustrik
|
yep
|
|
[12:25] sustrik
|
generic networking helpers
|
|
[12:25] sustrik
|
and the patterns on top of them
|
|
[12:26] locklace
|
that would also allow for developer modularity, e.g. so you can have different sets of people working somewhat independently on the different layers and progress them in parallel
|
|
[12:26] guido_g
|
utopia here we come
|
|
[12:26] locklace
|
basically this is just classic good stack/layering/protocol design
|
|
[12:27] sustrik
|
i was never interested in ACE/Boost.Asio side of the thing
|
|
[12:27] sustrik
|
but it seems it would be helpful to have such layer
|
|
[12:27] sustrik
|
you may try to separate the two
|
|
[12:27] sustrik
|
but it's not a trivial task
|
|
[12:28] locklace
|
well, since zmq already goes halfway or more toward giving the world a massively improved socket layer, it would be nice to see it evolve fully into that role...
|
|
[12:28] sustrik
|
hi guido :)
|
|
[12:28] sustrik
|
sure, give it a try
|
|
[12:28] guido_g
|
hi sustrik
|
|
[12:29] sustrik
|
locklace: one problem you'll encounter is that the 0mq API niceness
|
|
[12:29] sustrik
|
is a product of messaging patterns
|
|
[12:29] guido_g
|
the world is full of projects with a broad and _general_ scope... unsurprisingly most of them are dead
|
|
[12:29] guido_g
|
one of the best points of ømq is it's focus
|
|
[12:29] sustrik
|
by selecting a pattern, 0mq can do a lot of logic for you and never bothers you about it
|
|
[12:30] sustrik
|
when you start exposing all the details you need for generic messaging via API you end up with something very similar to BSD socket API
|
|
[12:30] sustrik
|
guido_g: well, that's my focus
|
|
[12:30] locklace
|
so you'd end up with a complete and well-defined socket layer and api (effectively a new network+transport layer that treats classic L4 circuits as its links, and allows you to request different kinds of delivery semantics), and then a layer of "messaging patterns" on top of that which simply crystallise the most common use cases, but which can then easily be extended
|
|
[12:31] sustrik
|
people are free to to try their own ideas
|
|
[12:31] sustrik
|
but somehow that doesn't happen :)
|
|
[12:31] guido_g
|
what a surprise
|
|
[12:31] sustrik
|
:)
|
|
[12:32] sustrik
|
locklace: try to do that; but recall the main motivation was that BSD socket API sucks
|
|
[12:32] sustrik
|
so if you end up with equivalent or worse API, you've failed to achieve the goal
|
|
[12:33] locklace
|
you could be right. it's certainly plausible that attempting to generalise zmq would be a regression and not an improvement. there's no question it would be tricky, but since it's already gone 3/4 of the way there or so, it doesn't look impossible to me
|
|
[12:34] locklace
|
i guess what i'd like to know is what the plans are for the future in this regard
|
|
[12:34] sustrik
|
no plans myself, but watch for discussion on the mailing list
|
|
[12:34] sustrik
|
it seems to be a hot topic :0
|
|
[12:34] sustrik
|
:)
|
|
[12:35] locklace
|
yeah
|
|
[12:35] locklace
|
when i see people grappling with classic network-layer problems like how to handle addressing and routing, then this where my thinking leads :)
|
|
[12:37] sustrik
|
well, i'm coming from an enterprise-messaging sphere
|
|
[12:37] sustrik
|
so i have enough experience with duplicit functionality
|
|
[12:38] locklace
|
heh
|
|
[12:38] sustrik
|
0mq actually evolved from classic enterprise messaging ideas
|
|
[12:38] sustrik
|
by removing the duplicities
|
|
[12:38] sustrik
|
almost everything didn't passed the duplicty test
|
|
[12:38] sustrik
|
and was removed
|
|
[12:38] locklace
|
i'm coming from core networking
|
|
[12:38] sustrik
|
i've figured out
|
|
[12:38] sustrik
|
what remained was messaging patterns
|
|
[12:39] sustrik
|
as those cannot be delegated to upper layer (like transactions or serialisation)
|
|
[12:39] sustrik
|
or to lower layer, like L3/L4
|
|
[12:40] locklace
|
i think that's a good insight
|
|
[12:40] sustrik
|
locklace: what's exactly is you area of experience?
|
|
[12:40] sustrik
|
i'm kind of missing lower layer people in 0mq community
|
|
[12:41] locklace
|
it just looks to me like zmq blurs that layer with what's effectively a "higher" L3/L4 set of functionality that's required to handle routing and reliability semantics. and what i see from present discussions is that it's exactly this area that's giving you trouble at the moment, and it's happening because the layers are blurred
|
|
[12:42] locklace
|
sustrik: core network architecture and platform/protocol design
|
|
[12:42] sustrik
|
are you involved in ietf by chance?
|
|
[12:42] locklace
|
yes
|
|
[12:42] sustrik
|
nice
|
|
[12:43] sustrik
|
been in prague?
|
|
[12:43] locklace
|
very recently :)
|
|
[12:43] sustrik
|
heh
|
|
[12:43] sustrik
|
i've posted a notice on the mailing list about me being in prague...
|
|
[12:43] sustrik
|
we could have had a beer
|
|
[12:44] locklace
|
yeah, i didn't actually know about zmq until about two days ago
|
|
[12:44] sustrik
|
pity
|
|
[12:44] sustrik
|
anyway, my idea is to get patterns standardised in the future
|
|
[12:44] sustrik
|
but there's a long way to go still
|
|
[12:45] locklace
|
standardised how?
|
|
[12:45] sustrik
|
hard to say
|
|
[12:45] sustrik
|
one option is
|
|
[12:45] sustrik
|
to start with fixing the semantics
|
|
[12:46] sustrik
|
and providing that as a means for bridging existing messaging implementations into larger topologies
|
|
[12:46] sustrik
|
the other option is to actually focus on the actual wire level protocol
|
|
[12:47] sustrik
|
but that's kind of contentious, given there's a lot of existing messaging protocols out there
|
|
[12:48] locklace
|
yes, it will be
|
|
[12:48] sustrik
|
anyway, i think there'll be a lot of support for that kind of thing in the future
|
|
[12:49] sustrik
|
the need for hardware-software interaction can even lead to need for actual binary protocol
|
|
[12:49] locklace
|
but i'd still suggest it's the right way to do things; it actually doesn't matter whether X group of people think it's contentious, what matters is the discipline of writing it up in that form and subjecting it to the full community review process that enables
|
|
[12:50] sustrik
|
yep
|
|
[12:50] sustrik
|
what's your email btw?
|
|
[12:50] locklace
|
i.e. if you can write the whole protocol up in draft form so it can be properly reviewed, you'll be able to emerge with a much better and more solid end result
|
|
[12:50] sustrik
|
i'll ping you if anything interesting happens in this area
|
|
[12:51] locklace
|
ok. i'll send you a note offline
|
|
[12:51] sustrik
|
thanks
|
|
[12:51] sustrik
|
souls*
|
|
[12:52] locklace
|
yeah, but there's a lot of work needed to even get near that point
|
|
[12:52] sustrik
|
ack
|
|
[12:52] locklace
|
the write-up above and an initial public review cycle, mainly ;)
|
|
[12:53] sustrik
|
yep, well get there
|
|
[12:54] locklace
|
if you (or someone with the knowledge of zmq) is willing to put the work in to write it all up in draft form, i might be able to help as editor and with some process stuff
|
|
[12:54] mikko
|
sustrik: http://sourceforge.net/projects/udt/forums/forum/393036/topic/4061830
|
|
[12:54] locklace
|
i think you'll also find that once you have it down in that format, it will greatly improve the traction you get on design issues on the mailing list etc.
|
|
[12:55] sustrik
|
mikko: i've seen that
|
|
[12:55] locklace
|
traction is always *much* easier to achieve when people can comment on specific parts of draft text
|
|
[12:55] sustrik
|
there doesn't seem to be an answer
|
|
[12:55] sustrik
|
other than "coming soon"
|
|
[12:56] sustrik
|
anyway, let's see, maybe the guy will really implement it
|
|
[12:56] sustrik
|
locklace: yes, you are right
|
|
[12:57] sustrik
|
the biggest problem with an initial draft is who to focus it on
|
|
[12:57] sustrik
|
enterprise messaging devs are not participating on ietf
|
|
[12:57] sustrik
|
while ietf folks have no idea of enterprise messaging
|
|
[12:57] sustrik
|
and, additionally, 0mq-style messaging is no longer an enterprise messaging proper
|
|
[12:58] locklace
|
well, my suggestion is to forget about targeting it at any specific group at the moment, but write it up in the format of an ietf draft
|
|
[12:58] locklace
|
regardless of where it goes from there, that will be a major step in the right direction even just internally for the zmq project and mailing list
|
|
[12:58] sustrik
|
yeah, it should probably be worded in the most generic way
|
|
[12:59] sustrik
|
so that anyone can understand it
|
|
[12:59] sustrik
|
but then, it would probably look like a vapourware
|
|
[12:59] sustrik
|
oh my
|
|
[12:59] locklace
|
yes, and writing it up in that format (a) helps you structure it as a proper protocol architecture, and (b) helps focus discussion
|
|
[13:01] sustrik
|
ack
|
|
[13:02] locklace
|
and of course, if it then does get edited into a shape fit for possible ietf work, that enables you to think about stuff like having a bof with a nonzero chance of success
|
|
[13:03] locklace
|
this is probably obvious, but i'll say it anyway: the first thing the draft should address is the set of requirements. in fact, the first version should probably be nothing *but* a set of requirements, which people can then argue over and loosely agree on before anything else is done
|
|
[13:04] sustrik
|
yes, that's the plan
|
|
[13:04] sustrik
|
the another interesting question is which area it should be dealt with in
|
|
[13:05] sustrik
|
so far i am in contact with apps area guys
|
|
[13:05] locklace
|
area?
|
|
[13:05] sustrik
|
ietf area
|
|
[13:05] sustrik
|
but the think is a bit lower layer
|
|
[13:05] sustrik
|
somewhere between transport and apps
|
|
[13:05] locklace
|
well let's just say there are good reasons why i said not to worry about targeting any particular group at the outset ;)
|
|
[13:06] sustrik
|
:)
|
|
[13:06] locklace
|
i really would not worry about that kind of thing at present, you have enough work to do without worrying about the organisational politics
|
|
[13:06] locklace
|
get a sound technical draft put together first, worry about where it goes to later
|
|
[13:07] locklace
|
it will be pretty much impossible to answer the "where should it go" question until the requirements and architecture are clear anyway
|
|
[13:09] locklace
|
xml2rfc makes writing things up easy now, yay
|
|
[13:09] sustrik
|
yep, i want to learn to use it
|
|
[13:09] locklace
|
it will take you five minutes
|
|
[13:09] sustrik
|
formatting drafts by hand is a pain
|
|
[13:10] locklace
|
just go to xml.resource.org, submit the xml and it spits out text. the actual xml schema is almost trivial, hit the "unofficial successor" link at the top of the page to get to the reference, and start with some existing draft .xml file as a template
|
|
[13:11] sustrik
|
thx
|
|
[13:44] pantsman
|
hi - it's great that there are windows binaries for pyzmq on its github downloads page, but I still need visual studio installed to run them - does this mean they're debug builds?
|
|
[13:45] sustrik
|
why do you need msvc?
|
|
[13:47] pantsman
|
sustrik, because without, the libzmq.dll that comes with the pyzmq installer can't find a DLL called Microsoft.VC90.DebugCRT
|
|
[13:48] sustrik
|
yes, the library was compiled with debug CRT
|
|
[13:49] sustrik
|
where have you got the binary from?
|
|
[13:49] pantsman
|
I was using https://github.com/downloads/zeromq/pyzmq/pyzmq-2.1.4.win32-py2.7.msi
|
|
[13:52] sustrik
|
no idea where that comes from
|
|
[13:52] sustrik
|
mikko: are you there?
|
|
[13:54] sustrik
|
hm, looks like he's not here
|
|
[13:54] sustrik
|
pantsman: can you please complain about it on the mailing list?
|
|
[13:55] pantsman
|
sustrik, sure
|
|
[13:55] sustrik
|
thanks
|
|
[13:56] guido_g
|
mentioning pyzmq in the subject, as it seems to be a problem with pyzmq build
|
|
[13:58] mikko
|
sustrik: yes
|
|
[13:58] NikolaVeber
|
sustrik, I have linked the post at http://www.zeromq.org/area:results,
|
|
[14:04] sustrik
|
mikko: any idea where that binary comes from?
|
|
[14:05] sustrik
|
NikolaVeber: thanks
|
|
[14:05] NikolaVeber
|
np! :)
|
|
[14:08] evax
|
NikolaVeber: the last test is fine, notice the 1e7 on top of the graph :)
|
|
[14:08] NikolaVeber
|
oh, yeah! :)
|
|
[14:09] NikolaVeber
|
I'll update the note
|
|
[14:10] NikolaVeber
|
done
|
|
[14:10] sustrik
|
btw, the order of graphs is somewhat chaotic
|
|
[14:14] NikolaVeber
|
just done that
|
|
[14:18] mikko
|
sustrik: the .msi?
|
|
[14:18] sustrik
|
yes
|
|
[14:19] mikko
|
must be minrk / bgranger
|
|
[14:19] sustrik
|
i see
|
|
[14:19] mikko
|
i would imagine
|
|
[14:19] mikko
|
Microsoft.VC90.DebugCRT
|
|
[14:19] mikko
|
sounds like it's a debug build
|
|
[14:19] mikko
|
rather than a release build
|
|
[14:19] sustrik
|
or a bad CRT option passed to the compiler
|
|
[14:20] sustrik
|
anyway, it's reported on the ML
|
|
[14:20] sustrik
|
it's up to pyzmq guys noqw
|
|
[14:32] pantsman
|
fortunately not a big problem for me since I can build libzmq.dll myself; however building pyzmq is something that's beyond me at the moment
|
|
[14:48] yawniek
|
if i add zmq.SNDMORE with pyzmq the receiver does not get the message imediately but only after the last message of the multipart message has been sent off. shouldnt it be possible to stream multipart messages?
|
|
[14:49] yawniek
|
err mybad
|
|
[14:57] sustrik
|
yawniek: it's transactional
|
|
[14:57] sustrik
|
message is treated as an atomic unit
|
|
[14:58] sustrik
|
you either get whole message, or none of it
|
|
[14:59] yawniek
|
ok. i want to build an interface for a database where the results should be streamable. so query gets sent and a stream of messages should come back. how should the architecture look like?
|
|
[15:00] sustrik
|
the reply doesn't fit into memory, right?
|
|
[15:01] yawniek
|
exactely
|
|
[15:01] sustrik
|
0mq holds messages in memory
|
|
[15:02] sustrik
|
so it's hard to work with a message that does not fit into it
|
|
[15:02] sustrik
|
you can use XREQ/XREP to get a streaming like behaviour
|
|
[15:02] sustrik
|
not nice, but doable
|
|
[15:02] sustrik
|
check the guide
|
|
[15:04] yawniek
|
yeah, im reading it over and over again. i think a two step process with PUSH/PULL socket for the streaming is a way but that gives some overhead
|
|
[17:51] yziquel
|
is there any quick and fast way of knowing the ip of the host of a client socket that connects on a server? from the server?
|
|
[17:56] erickt
|
yziquel: I don't believe that's exposed from the api
|
|
[17:56] yziquel
|
ok. thanks.
|
|
[17:58] erickt
|
you can get the identity for certain socket types. That's talked about here: http://zguide.zeromq.org/page:all
|
|
[18:04] yziquel
|
what i really need is a directory of endpoints which is info to be served by a central server.
|
|
[18:04] yziquel
|
erickt: so i'm wondering how to populate it precisely.
|
|
[18:05] mikko
|
yziquel: there is a project aiming to do that
|
|
[18:06] mikko
|
yziquel: but i think it's only a wiki-page has been written about it this far
|
|
[19:01] mpales
|
hi
|
|
[19:01] mpales
|
i have a question related to application design
|
|
[19:02] mpales
|
anyone here?
|
|
[19:03] isvara
|
I am here, but that probably doesn't help you.
|
|
[19:09] sustrik
|
mpales: hi
|
|
[19:10] mpales
|
sustrik: i am trying to create a messaging pattern and need a help
|
|
[19:11] mpales
|
the system should reassemble segmented messages and process them
|
|
[19:11] mpales
|
i'd like to have several workers for reassembling
|
|
[19:12] mpales
|
the point is that is a worker gets a segment, it should also receive all thge other segments
|
|
[19:12] mpales
|
so i cannot use a plain round robin or lru
|
|
[19:12] sustrik
|
why not multipart messages?
|
|
[19:13] mpales
|
the application has no control over segments
|
|
[19:13] mpales
|
they are the input
|
|
[19:13] mpales
|
other clients sent them
|
|
[19:14] sustrik
|
why don't they sent them as multipart msgs?
|
|
[19:14] mpales
|
it's not a zmq based system
|
|
[19:15] sustrik
|
ok, what's the problem with 0mq then?
|
|
[19:15] mpales
|
how to route those segments
|
|
[19:15] sustrik
|
i mean
|
|
[19:15] sustrik
|
you get a list of segments
|
|
[19:15] mpales
|
all segments with same id should go to the same worker
|
|
[19:15] sustrik
|
then you send it to a worker as a multipart message
|
|
[19:17] mpales
|
but all segments belong to the same message
|
|
[19:17] sustrik
|
?
|
|
[19:17] sustrik
|
i think i don't get it
|
|
[19:17] mpales
|
oops, my mistake
|
|
[19:18] mpales
|
segments belong to different messages
|
|
[19:18] guido_g
|
create a map: source id -> ømq worker
|
|
[19:18] sustrik
|
aha
|
|
[19:18] sustrik
|
you want stateful services
|
|
[19:19] mpales
|
guido: that's also my idea
|
|
[19:19] mpales
|
sustrik: yes
|
|
[19:19] sustrik
|
there's no easy way to do that in 0mq
|
|
[19:19] sustrik
|
the issue is being hotly discussed
|
|
[19:19] guido_g
|
mpales: look at the lru examples in the guide
|
|
[19:20] sustrik
|
yep, have a look into the guide
|
|
[19:20] guido_g
|
they will show you how to deal w/ socket identities
|
|
[19:20] sustrik
|
there are various workarounds there
|
|
[19:20] mpales
|
sustrik: i actually do not really expect that zmq will support it, but want to be sure :)
|
|
[19:20] guido_g
|
and instead of the lru use the source id for routing
|
|
[19:20] mpales
|
i already read the whole guide
|
|
[19:21] guido_g
|
so?
|
|
[19:21] mpales
|
sustrik: since the routing decision is on application level and zmq sits below it
|
|
[19:21] sustrik
|
what pieter does in the guide is using XREP socket as a way to route message to specific peers
|
|
[19:22] mpales
|
guido_g: yes, that could do it
|
|
[19:22] sustrik
|
that kind of helps with stateful services
|
|
[19:22] guido_g
|
mpales: it will because your problem is only a variation of the lru example
|
|
[19:22] sustrik
|
it basically shifts all the responsibility for routing to the user
|
|
[19:23] mpales
|
ok, thanks guys
|
|
[19:24] mpales
|
sustrik: the only problem with using xrep i have that it discards messages if they cannot be delivered
|
|
[19:24] mpales
|
in my case i woulk prefer blocking
|
|
[19:24] mpales
|
if the worker is busy
|
|
[19:25] mpales
|
since i _need_ to deliver to that particular worker and prefer to wait over dropping the message
|
|
[19:25] sustrik
|
mpales: yes, that's because xrep is meant to be a stateless service
|
|
[19:25] sustrik
|
ie. it sends the reply
|
|
[19:25] sustrik
|
if the requester is not there anymore it drops the message
|
|
[19:26] mpales
|
sustrik: yes, that's clear
|
|
[19:26] mpales
|
what i need is a xrep alternative that will block
|
|
[19:26] sustrik
|
so, i would say, you have to modify 0mq itself
|
|
[19:26] sustrik
|
introduce the blocking behaviour
|
|
[19:27] mpales
|
sustrik: i was thinking about a socket type where you can configure its behaviour - to avoid multiple types
|
|
[19:28] sustrik
|
if you believe you can deal with all the possible combinations, why not
|
|
[19:28] guido_g
|
sustrik: would zmq_send return != 0 in such a case?
|
|
[19:29] guido_g
|
i mean when sending to a xrep socket w/ hwm reached
|
|
[19:29] sustrik
|
with ZMQ_NOBLOCK flag it returns -1 and EAGAIN
|
|
[19:29] sustrik
|
without it, it just blocks
|
|
[19:29] guido_g
|
ha! :)
|
|
[19:30] guido_g
|
no, i was talking of xrep, where it drops the message
|
|
[19:30] sustrik
|
no, it's silent
|
|
[19:31] guido_g
|
what if this would be changed?
|
|
[19:31] sustrik
|
mpales: would that help?
|
|
[19:31] sustrik
|
sounds easy to implement
|
|
[19:31] guido_g
|
it still drops the message but the return code signales that the send was *not* done
|
|
[19:31] mpales
|
the configuration of that 'generic' socket could be limited to incoming/outgoing routing strategy and hwm option action
|
|
[19:31] guido_g
|
like with xreq in NOBLOCK mode
|
|
[19:33] mpales
|
sustrik: you're suggesting to set ZMQ_NOBLOCK to the socket?
|
|
[19:34] sustrik
|
nope, i was just answering guido_g
|
|
[19:34] guido_g
|
mpales: the idea was to make the send on an XREP socket retunr != 0 when the message has been droppen, so that the application has a way to see that the message was not sent
|
|
[19:35] mpales
|
guido_g: aha, that would help
|
|
[19:36] sustrik
|
mpales: check zmq::xrep_t::xsend
|
|
[19:36] sustrik
|
look for this line: if (it != outpipes.end ()) {
|
|
[19:37] mpales
|
sustrik: got it
|
|
[19:38] guido_g
|
sustrik: any reasons that this should not be done for all socket types that drop silently on send?
|
|
[19:39] sustrik
|
this is just a workaround for mpales
|
|
[19:39] sustrik
|
in reality it doesn't really woek
|
|
[19:39] sustrik
|
work
|
|
[19:39] guido_g
|
why?
|
|
[19:39] sustrik
|
if you insert a device between sender and receiver, the message may be dropped without sender ever noticing it
|
|
[19:40] guido_g
|
i see
|
|
[19:41] sustrik
|
it's a "scalability check" principle
|
|
[19:41] sustrik
|
should be documented somewhere even
|
|
[19:42] guido_g
|
need heavy re-wiring of some brain areas
|
|
[19:42] sustrik
|
:)
|
|
[19:43] guido_g
|
i'm playing w/ ømq for months now and still "don't get it"
|
|
[19:45] sustrik
|
i tend to play "what if it was TCP/IP?" game
|
|
[19:45] sustrik
|
if you add a router into middle, would it still work?
|
|
[19:46] guido_g
|
yes
|
|
[19:46] guido_g
|
it's just that i need to develop this as a reflex
|
|
[19:47] guido_g
|
it'S a matter of training i guess
|
|
[19:47] sustrik
|
yep, i'm training it for couple of years now
|
|
[19:47] sustrik
|
still i sometimes get trapped
|
|
[19:47] guido_g
|
*sigh*
|
|
[19:49] mpales
|
sustrik: in that code i also need to change the logic to be able to do a resend, right?
|
|
[19:50] sustrik
|
well, you should not change the state of socket in any way if you are dropping the message
|
|
[19:50] sustrik
|
the question is what to do with subdequent message parts
|
|
[19:51] sustrik
|
in case of multipart messages
|
|
[19:51] sustrik
|
ie. first one is dropped
|
|
[19:51] sustrik
|
do we expect other parts yet to be sent
|
|
[19:51] sustrik
|
or do we expect the user to back down and don't send subsequent parts?
|
|
[19:52] mpales
|
sustrik: i would suggest the latter
|
|
[19:52] sustrik
|
it's up to you
|
|
[19:53] sustrik
|
do it in such a way as to make you app benefit from it
|
|
[19:56] mpales
|
sustrik: is it possible that only first part is sent and the rest is dropped in case the queue is full?
|
|
[19:56] sustrik
|
no
|
|
[19:56] sustrik
|
it's atomic
|
|
[19:56] sustrik
|
it either sends the whole message
|
|
[19:56] sustrik
|
or nothing
|
|
[19:57] mpales
|
ok
|
|
[20:01] mpales
|
sustrik: i also noticed that ypipe is affected by false sharing in case you have several io threads or several contexts
|
|
[20:02] mpales
|
putting cacheline size pads between *w and *r would help
|
|
[20:02] sustrik
|
nice spot
|
|
[20:02] sustrik
|
would you provide a patch?
|
|
[20:03] mpales
|
i can, sure
|
|
[20:03] sustrik
|
:)
|
|
[20:03] sustrik
|
great
|
|
[20:03] mpales
|
but later this week, too busy these days
|
|
[20:43] private_meta
|
When using IPC sockets, how do I specify absolut locations of IPC files?
|
|
[20:43] private_meta
|
+e
|
|
[20:48] sustrik
|
ipc:///my/path
|
|
[20:49] private_meta
|
yeah, found it as well, thanks
|
|
[21:27] eyeris
|
I've installed zeromq 2.1.4 under cygwin but gcc complains about undefined references when I compile a simple test program (from the guide).
|
|
[21:28] eyeris
|
all of the undefined references it complains about begin with _zmq
|
|
[21:30] Seta00
|
are you linking your project with zmq properly?
|
|
[21:30] eyeris
|
Yeah, making a paste now
|
|
[21:31] eyeris
|
http://pastebin.ca/2046149
|
|
[21:34] eyeris
|
If I run make check in the zeromq build tree a bunch of them fail and then it seems to hang after test_hwm.exe passes
|
|
[23:27] stevan
|
is this a good place to ask about jzmq build issues on windows?
|
|
[23:28] stevan
|
or is there a more jzmq specific room?
|
|
[23:28] stevan
|
this is what I am getting -> http://codepeek.com/paste/4da6318a6cac63f86c754600
|
|
[23:28] stevan
|
libzmq compiled just fine
|
|
[23:29] stevan
|
this is with Visual Studio 2010 too, if that makes any difference
|
|
[23:52] stevan
|
ah ha, I got it
|
|
[23:53] stevan
|
libzmq HEAD was not what I really wanted, I wanted the one on the downloads page
|
