IRC Log

Monday March 21, 2011

[Time] NameMessage
[09:54] pieterh sustrik: ping
[10:01] pieterh ok, new packages of 0MQ 2.0 and 2.1 are fresh and hot!
[10:05] Guthur pieterh: should all socket types be able to connect and bind in any order?
[10:05] pieterh Guthur: in an ideal world, yes, but there are gaps
[10:05] pieterh e.g. PAIR sockets can't
[10:06] Guthur what about PUSH/PULL
[10:06] pieterh anything over inproc can't either
[10:06] Guthur ah it was the inproc
[10:06] pieterh yep
[10:06] pieterh with inproc the pattern is 'bind, start child thread, which connects'
[10:06] Guthur I suppose inproc ain't so bad, relatively easy to control the order
[10:07] pieterh since it's structurally different from other transports, I use it only when I'm doing literal multithreading
[10:07] pieterh as compared to multitasking with threads... which I use tcp or ipc for
[10:07] pieterh btw I fixed the guide examples to fit 73 columns, thanks for that hint
[10:07] Guthur you're welcome
[10:08] Guthur it wasn't too bad, but you had set the bar high so these wee things are noticeable, hehe
[10:09] Guthur Ian Barber's PHP conf video is really good
[10:10] pieterh the bar is never too high :-)
[10:10] pieterh Ian Barber is a phenomenon, it was great to meet him in London last week
[10:11] pieterh but then, this community seems to attract bru
[10:11] pieterh *brilliant people
[10:11] Guthur The mention of stable/unstable components was very enlighting
[10:11] Guthur static/dynamic is the other way of describing it
[10:13] Guthur His highlevel description of a imaging process system was a inspirational as well, fanning out and in where appropriate
[10:13] Guthur I'm looking a similar architecture for a 0MQ FIX engine
[10:13] Guthur a/at
[10:14] pieterh yup, understanding the stable/unstable aspects of an architecture is key to making it right
[10:21] Guthur I find myself now looking at my arch and asking the question is this a stable/unstable component
[10:59] sustrik pieterh: pong
[11:00] pieterh sustrik: I was just wondering if it was OK to mention the plans for 3.0 to zeromq-dev
[11:00] sustrik yes
[11:00] sustrik i'm going to do that now
[11:00] pieterh goodo
[11:01] pieterh I think it's important to say that we will maintain 2.1 for a long time
[11:01] pieterh change is fun but also scary to anyone wanting to use 0MQ in production
[11:02] sustrik yep, several years presumably
[11:02] pieterh well, I was thinking of 6-12 months, more like :-)
[11:02] sustrik although the impact of new API won't be that bad as it will affect only C apps
[11:02] pieterh I assume we will have a 3.0 stable fairly rapidly
[11:02] pieterh indeed
[11:03] Guthur sustrik:why will the API change only effect C apps?
[11:03] pieterh if we do the API changes separately from semantic changes like xpub/xsub, it's less painful
[11:03] pieterh Guthur: also affect all bindings, of course
[11:03] sustrik Guthur: have a look at 3.0 roadmap
[11:03] sustrik you'll see that most changes don't affect the bindings API
[11:03] sustrik although it obviously affect binding implemetations
[11:04] Guthur oh the roadmap page has been updated
[11:04] Guthur I had not noticed that
[11:04] sustrik yes, i've listed all the backward imcompatible changes there
[11:04] pieterh sustrik: I'm going to already switch to the new ROUTER/DEALER terminology in the guide
[11:04] sustrik feel free to comment
[11:04] pieterh this pushes that change up
[11:04] sustrik ok
[11:05] sustrik pieterh: yes, 2 steps
[11:05] sustrik 1, change the API --> do as quickly as possible
[11:05] pieterh sustrik: I'd like to get a stable 3.0 with new API and no real functionality changes, so we can kill 2.0
[11:05] pieterh yes
[11:05] sustrik 2. add new functionality
[11:05] pieterh yes, +1
[11:06] pieterh I think the new release process is working nicely, it's certainly easy
[11:07] sustrik well, nobody have really tested the recent changes
[11:07] sustrik let's hope it works as expected
[11:08] Guthur what needs to be done to zmq_send/recv to make them coherent with POSIX?
[11:08] sustrik zmq_send (void *data, size_t size, int flags);
[11:08] sustrik same for zmq_recv
[11:08] sustrik no zmq_msg_t
[11:08] pieterh sustrik: I've been testing pretty heavily, via Guide examples
[11:08] sustrik ah, ok then
[11:09] Guthur and then they would wrap it as a zmq_msg internally?
[11:09] sustrik yes
[11:09] Guthur umm, that's pretty nice
[11:09] sustrik the original signature will still be available as zmq_sendmsg and zmq_recvmsg
[11:09] pieterh sustrik: one thing I noticed is missing is zero-copy on recv
[11:09] sustrik yes?
[11:09] Guthur that could actually simplify bindings
[11:09] pieterh I assume this is what we can do with 3.0
[11:10] pieterh size = zmq_recv (socket, void **data)
[11:10] Guthur *authoring bindings
[11:10] pieterh where I get a malloc'd blob I can then pass around
[11:10] sustrik can't be done
[11:11] sustrik because you don't know what allocation mechanism was used to allocate the buffer
[11:11] sustrik thus, you don't know whether to free it or what
[11:11] pieterh ah, I mean recv() would always allocate on the heap
[11:11] sustrik copying the data?
[11:11] pieterh no copying, allocate a block, read into that block, and provide the caller with the block address
[11:11] pieterh the caller then uses the block with no copying
[11:12] pieterh today, you have to always copy the data received
[11:12] sustrik the socket is not in control of the allocation process
[11:12] sustrik it can be say another socket allocating buffer
[11:12] sustrik and sending it via inproc
[11:12] pieterh inproc is special...
[11:12] sustrik it is
[11:12] pieterh forgetting inproc for a second...
[11:13] sustrik you still have the VSM problem
[11:13] sustrik very small messages are allocated on the stack
[11:13] sustrik to speed the processing
[11:13] pieterh in C?
[11:13] sustrik inside libzmq
[11:13] pieterh how do you return a zmq_msg_t referring to a VSM on the stack?
[11:13] sustrik it's embedded
[11:14] sustrik see the definition in zmq.h
[11:14] pieterh ok, sure
[11:14] sustrik in short, there are different alloc/dealloc mechanisms
[11:14] sustrik zmq_msg_t cleanly wraps the buffer and the deallocation function
[11:14] pieterh ok, at the risk of being glib, forget VSMs for a second too, they're special
[11:14] sustrik ok
[11:15] pieterh the bulk case is large messages, where copying is a cost
[11:15] Guthur so inproc is special and VSM is special
[11:15] pieterh yes, this is accurate, they're both optimizations
[11:15] pieterh which means they can depend on a more complex API, that's fine
[11:16] pieterh sustrik: the semantic which I used in the zmsg class, and which works really neatly
[11:16] sustrik if you don't care about allocation and copying cost in inproc and small message scenarios
[11:16] Guthur at the moment though this specialness is too apparent the user, the inproc issue earlier withstanding
[11:16] pieterh is that recv() returns a freshly allocated block, and send() always frees a block when it's done
[11:16] Guthur iis/isn't
[11:16] sustrik you can wrap zmq_sendmsg to do the transformation of zmq_msg_t into alloc'd buffer
[11:17] pieterh sustrik: I do care, but we can (will) make multiple levels of API
[11:17] sustrik and what's the point of the proposal?
[11:18] pieterh hmm, well,.. :-)
[11:18] pieterh zero-copy on read
[11:18] pieterh in the simplest possible fashion
[11:18] pieterh ok, forget it, POSIX socket API doesn't offer this
[11:18] sustrik while at the same time sacrificing zero copy in some scenarios
[11:18] sustrik zero-copy is a beast
[11:18] pieterh as long as you can recv() directly into a user-allocated buffer, you get zero copy
[11:18] sustrik no support in POSIX
[11:19] pieterh I just want to be able to store message contents without copying them
[11:19] sustrik yes, but that makes the messaging lock-step
[11:19] pieterh indeed
[11:19] pieterh only the recv knows how much memory to allocate
[11:19] sustrik exactly
[11:20] sustrik so think of zmq_msg_t as a buffer
[11:20] pieterh or else, as has been discussed, a user-pluggable allocator
[11:20] pieterh which is horrid IMO
[11:20] sustrik yuck
[11:20] sustrik you can use tcmalloc and similar though
[11:21] sustrik as they expose alloc/free API
[11:21] pieterh they look like malloc/free
[11:21] sustrik you just link the library
[11:21] pieterh yup
[11:21] sustrik and all allocations are redirected
[11:21] pieterh anyhow, think about it, if there's a way to return a fresh buffer on recv
[11:21] pieterh it would be elegant IMO
[11:21] sustrik well, zmq_msg_t is a buffer
[11:21] sustrik it looks something like this:
[11:21] pieterh yeah, but it's not a buffer
[11:22] sustrik struct zmq_msg_t
[11:22] sustrik {
[11:22] sustrik void *data
[11:22] pieterh it's a structure that contains a buffer
[11:22] sustrik size_t size;
[11:22] sustrik free_fn_t free_fn;
[11:22] sustrik }
[11:22] sustrik ;
[11:22] pieterh well...
[11:22] pieterh ok, point taken, it does work
[11:22] pieterh it's a blob descriptor
[11:22] sustrik yes
[11:23] pieterh (a) it should be called 'blob_t' and (b) recv() should return a freshly created blob
[11:23] pieterh blob_t *blob = zmq_recv (socket)
[11:23] sustrik recv has to return int
[11:23] sustrik to be POSIX compliant
[11:23] sustrik int + errno
[11:23] pieterh int rc = zmq_recv (socket, &blob)
[11:23] sustrik yes
[11:23] sustrik that's how it works now
[11:23] pieterh point being, recv creates the buffer
[11:24] sustrik except for the name
[11:24] pieterh no, now you need this extra init
[11:24] sustrik ah
[11:24] pieterh rather annoying
[11:24] sustrik that's to avoid additional malloc
[11:24] pieterh optimization
[11:24] sustrik so you create blob_t on the stack
[11:24] sustrik which is really fast
[11:24] pieterh I know you want that but it shouldn't be in the simple API
[11:24] sustrik instead of putting it on the heap
[11:25] pieterh simple API should be just fast enough, and as simple as it can be, and POSIX like
[11:25] pieterh IMO
[11:25] sustrik in 3.0 the simple API will be POSIX like
[11:25] sustrik zmq_send (void*, size_t)
[11:25] sustrik zero-copy is never neat
[11:25] sustrik it's optimisation in itself
[11:26] pieterh what's the zmq_recv syntax for 3.0...?
[11:26] sustrik and one that crosses all levels of stack
[11:26] sustrik and makes mess of layering
[11:26] pieterh int size = zmq_recv (socket, void *)...?
[11:26] sustrik zmq_recv (void *buf, size_t *size, int flags);
[11:26] sustrik forgor socket
[11:26] sustrik forgot
[11:26] sustrik same as POSIX recv()
[11:27] pieterh ok, should be 'ssize_t zmq_recv(void *socket, void *buf, size_t len, int flags);
[11:27] sustrik size_t *len
[11:27] pieterh nope
[11:27] pieterh size_t len, maximum size of buffer
[11:27] sustrik ah
[11:27] pieterh return code is actual size read
[11:28] sustrik you are right
[11:28] pieterh it's hugely annoying because you have to allocate randomly huge buffers
[11:28] pieterh and there's no way to recover if the buffer was too small
[11:28] pieterh conclusion: you *cannot* use the POSIX socket API 100% for recv
[11:28] sustrik EMSGSIZE
[11:28] pieterh no way to recover
[11:28] sustrik then you can increase the buffer
[11:28] sustrik and recv() anew
[11:28] pieterh 'please go back and re-read'?
[11:28] sustrik however
[11:28] pieterh ugh
[11:29] pieterh ugh and ugh
[11:29] sustrik in most scenarios you want to put the limit on message size anyway
[11:29] pieterh it means putting message lengths into separate frames
[11:29] sustrik setsockopt (MAXMSGSIZE, 1000);
[11:29] pieterh that's orthogonal to this
[11:29] sustrik char buff [1000];
[11:29] sustrik zmq_recv (buff, 1000);
[11:29] pieterh ugh and ugh again
[11:30] pieterh I have frames of 8 bytes and frames of 2MB
[11:30] pieterh seriously unsatisfactory to not be able to properly allocate the right buffer size for each frame
[11:30] sustrik then you need to use more complex API
[11:31] pieterh this makes the simple API almost useless IMO
[11:31] pieterh nice for send, useless for recv
[11:32] sustrik what's the alternative then?
[11:32] sustrik tha'ts how UDP works, irrc
[11:32] pieterh I don't know, but reading into a pre-allocated buffer is inacceptable IMO
[11:32] pieterh UDP has maximum frame sizes
[11:32] pieterh TCP is a stream
[11:32] sustrik ZMQ_MAXMSGSIZE
[11:33] sustrik same as MTU
[11:33] pieterh 0MQ is message-oriented, that means different read/write semantics
[11:33] sustrik but on messaging level
[11:33] pieterh it's not the same thing
[11:33] pieterh you have a 64-bit message size
[11:33] sustrik well, i think there are 2 distinct use cases here
[11:33] sustrik 1. something like market data
[11:34] pieterh I think it's necessary to define semantics that fit messaging
[11:34] sustrik you want to limit max size of message
[11:34] pieterh not try to make 0MQ look like TCP, it's not
[11:34] sustrik to prevent someone killing the transmission by pushing 4GB blob
[11:34] pieterh this is orthogonal
[11:34] pieterh it's not the issue
[11:34] sustrik in such case zmq_recv() is sufficient
[11:34] pieterh I can't agree...
[11:34] sustrik 2. something like FTP
[11:34] pieterh let's say I want to hold a queue of messages
[11:34] sustrik in that case you don't know the size in advance
[11:35] pieterh might be 1M messages, held for 1 second
[11:35] pieterh I have to allocate the maximum frame size for each one
[11:35] pieterh this is seriously problematic
[11:35] pieterh my alternative is to read everything into a frame buffer and memcpy
[11:35] pieterh that is seriously stupid
[11:35] pieterh what I want is a recv() that allocates and returns a block
[11:35] pieterh that is seriously simple
[11:35] sustrik that's zmq_recvmsg
[11:36] pieterh then I will never use recv() because it's *more* complex
[11:36] pieterh which is pointless
[11:36] sustrik ok
[11:36] pieterh trying to make 0MQ look like TCP is a big mistake IMO
[11:36] Guthur frankly from what I'm reading POSIX recv is just pants
[11:36] pieterh it is simply not a file-oriented metaphor, period
[11:36] Guthur and modelling it after that is the problem
[11:36] pieterh yes
[11:36] pieterh and it won't make 0MQ more compatible with the kernel
[11:37] sustrik so you want zmq_recvmsg and no zmq_recv, right?
[11:37] pieterh the way to get 0MQ into the kernel is to make it utterly compelling
[11:37] pieterh so every developer on the planet uses it
[11:37] pieterh and you get that by making it utterly simple for simple cases
[11:37] pieterh sustrik: what I want, speaking as a 0MQ user, is a recv method (any name) which returns a fresh buffer or blob structure
[11:38] sustrik zmq_recvmsg
[11:38] sustrik that's it
[11:38] pieterh and a send method that takes the same buffer or blob structure
[11:38] sustrik zmq_sendmsg
[11:38] pieterh sure
[11:38] pieterh then recv and send are useless
[11:38] sustrik it's going to be there
[11:38] pieterh and won't get showtime
[11:38] sustrik zmq_send (s, "ABC", 3);
[11:38] sustrik what's wrong about that?
[11:38] pieterh using one API metaphor for sending, another for receiving?
[11:39] sustrik char buff [3];
[11:39] sustrik zmq_recv (s, buf, 3)
[11:39] pieterh yes, and if the sender gives you 4 bytes, your socket is dead
[11:39] sustrik the sendmsg and recvmsg will be of course availabler
[11:39] pieterh it's not defensible
[11:39] sustrik it's not dead
[11:39] sustrik it reports error
[11:39] pieterh and the next read...
[11:39] pieterh next frame or same frame?
[11:40] sustrik i would copy UDP semantics
[11:40] pieterh or a special flag to re-read the same frame?
[11:40] sustrik but it's open to discussion
[11:40] pieterh it gets horrid
[11:40] pieterh and it's not worth it just to get that simple send
[11:40] pieterh I can make a 'send string' method in 10 lines of C, and have done this
[11:40] pieterh it's not worth upgrading to 3.0 to save 10 lines of C
[11:40] sustrik so you want to keep the existing API right?
[11:40] pieterh send is really not the issue
[11:41] pieterh nope, the existing API is also too complex
[11:41] pieterh sorry :-)
[11:41] sustrik ok, send your proposal to the mailing list
[11:41] sustrik no point in discussing it here
[11:41] pieterh well, I have thousands of lines of sample code in the Guide
[11:41] pieterh that show what I like as APIs, in C
[11:41] pieterh string = s_recv (socket); free (string);
[11:41] sustrik using blobs, yes
[11:42] pieterh s_send (socket, "Hello");
[11:42] pieterh that's for strings, it's simple and clean
[11:42] sustrik that's a special case of current 0mq api
[11:42] sustrik you can propose changing zmq_msg_t into blob_t
[11:42] pieterh anyhow, my advice is to not try to imitate neither TCP nor UDP but develop clear and optimal 0MQ semantics in the API
[11:43] pieterh you will get more adoption of your API
[11:43] pieterh and that's the way to get 0MQ into the kernel
[11:43] sustrik sure, go on, propose an alternative
[11:43] pieterh slavish imitation of old, semi-accurate APIs won't work
[11:43] pieterh ok, I'll propose an alternative
[11:43] sustrik ok
[11:44] sustrik as for kernel it's a very different problem so let's forget about it atm
[11:44] pieterh well, I think the point is that simple / clean gets you market share
[11:44] sustrik sure
[11:44] sustrik kernel is different though -- if you propose change to ABI, you'll never get in
[11:44] pieterh the kernel is just peopl;e
[11:45] pieterh *people
[11:45] sustrik the problem with new ABIs is they have to be maintained indefinitely
[11:45] Guthur tis a shame C doesn't support method overloading
[11:45] Guthur method/function
[11:45] pieterh Guthur: well, it's nice that you know what a function is, forever
[11:46] pieterh sustrik: do you mind if I delete the irrelevant comments on the 3.0 page?
[11:46] sustrik which are those?
[11:46] pieterh all of them, I guess
[11:46] sustrik ah, the comments
[11:46] sustrik yes
[11:46] sustrik sure
[11:46] sustrik i haven't known it's even possible
[11:46] Guthur overloading doesn't remove that but you can have Send(blob*, size) Send(zmq_msg) etc
[11:47] Guthur anyway not really benefiting the discussing
[11:47] sustrik Guthur: that helps with API, but doesn't help with ABI
[11:47] sustrik from ABI point of view it's still 2 functions
[11:48] Guthur but this was a discussion about API,
[11:49] Guthur i thought
[11:49] sustrik mostly, there are some ABI users though
[11:49] sustrik namely, clrzmq :)
[11:49] sustrik right?
[11:49] mile yrashk,
[11:50] Guthur well... yeah
[11:57] yrashk mile: ?
[11:58] mile yrashk, hi, I was just into connecting yaws to erlzmq2, and I'd need some advice
[11:59] yrashk mile: sure
[11:59] mile docs says it creates an OS thread per context
[12:00] yrashk that's right
[12:00] mile that means, if I want to serve a large number of connections (parallel), I should make some kind of a router talking to zmq
[12:00] mile which then talks to many erlang processes
[12:00] yrashk do you plan to open a large open of contexts?
[12:01] pieterh sustrik: ok, I've thrown together a very rough API proposal
[12:01] mile yes, a web server in the front
[12:01] sustrik ok
[12:01] mile so, the more, the better :)
[12:01] pieterh sustrik: mostly, there is a really nice class-oriented API style we've developed for ZFL
[12:01] Guthur pieterh: you sending to the mailing list, or is it on the wiki
[12:01] yrashk is there any good reason to open a large # of contexts?
[12:02] mile a good example is comet/ajax clients
[12:02] sustrik pieterh: can you send that to the ML
[12:02] pieterh Guthur: my pattern is to document on the wiki first, then tell the list or IRC about it
[12:02] pieterh sending specs / proposals to the ML is an anti-pattern
[12:02] Guthur no probs, just eager to have a look
[12:02] mile yrashk, no good reason, one would do good, it it takes care of multiplexing
[12:02] sustrik nobody is going to notice it
[12:02] yrashk mile: our recv does multiplex
[12:03] mile yrashk, so a single recv which then talks to a large number of erlang processes should work fine?
[12:03] pieterh sustrik: what are the actual goals with the 3.0 API?
[12:04] sustrik move towards POSIX
[12:04] pieterh simpler API?
[12:04] pieterh less lines of code in apps?
[12:04] sustrik nope, POSIX-like api
[12:04] pieterh that's all?
[12:04] sustrik for me, yes
[12:04] yrashk mile: well, you can spawn multiple recvs on the erlang side for different sockets (even in a single context, but you can do multiple contextrs too) and every context will multiplex (it uses zmq_poll)
[12:05] pieterh ok, I'll add my goals too, they're not in conflict afaics
[12:05] tjgillies anyone have a pub/sub example for node? i can't find one anywhere
[12:05] sustrik + removing some obsolete stuff
[12:05] sustrik but please use mailing list
[12:05] sustrik nobody will see the discussion on the webpage
[12:06] pieterh sustrik: like I said, I document on the wiki first, *then* discuss on the list
[12:06] mile yrashk, thanks! :) do you know of any code snippets around?
[12:06] sustrik ok
[12:06] mile I couldn't find much on the webpage
[12:06] yrashk mile: perf/ and test/
[12:06] yrashk mile: in the repo
[12:06] sustrik but i makes unclear what's your proposal
[12:06] mile tx :)
[12:06] sustrik at least mark it as such
[12:06] pieterh sustrik: have you read the page?
[12:06] sustrik yep
[12:07] pieterh it's literally a brain dump, not organized yet
[12:07] yrashk mile: yw
[12:07] sustrik whatever
[12:07] sustrik just move the discussion the mailing list
[12:07] pieterh Yes, I'm in the process of doing that
[12:07] sustrik goodo
[12:08] pieterh you'd prefer that my proposal is marked "proposed by Pieter" or somesuch?
[12:08] pieterh we don't generally do this
[12:08] pieterh wiki pages = edit into shape and consensus
[12:09] sustrik doesn't really matter as long as it's discussed on the mailing list
[12:09] sustrik then it's clear who proposed what
[12:13] pieterh ok, done
[12:13] Guthur will pushing string handling down into 0MQ have any adverse effect on people sending binary blobs
[12:13] pieterh Guthur: it's two different use cases
[12:13] pieterh sending strings is one very common case
[12:13] Guthur sure, but binary serialization is quite popular these days
[12:13] pieterh it's only in C that a string and a blob are fundamentally different
[12:14] Guthur protobuffers etc
[12:14] pieterh yeah, but this is not stuff you want to learn as a beginner
[12:15] Guthur if you are a beginner you probably shouldn't be using C
[12:15] Guthur you're feet can only take so much
[12:15] pieterh perhaps it was an error to use C as the language for the Guide
[12:16] Guthur pieterh: you have done a good job there though
[12:16] pieterh with a lot of help from contributors...
[12:16] Guthur I suppose my comment was more programming general as opposed to 0MQ beginners
[12:17] Guthur I'd imagine most of the bindings provide string handling
[12:18] Guthur I know i did with clrzmq
[12:18] pieterh perhaps we should make a standard C binding... :-)
[12:18] Guthur well you did mention multiple levels of the API
[12:20] pieterh hmm
[12:22] pieterh sustrik: if we made a separate C binding, most of my concerns could be solved there
[12:23] sustrik you mean like C binding on top of POSIX api
[12:23] pieterh yes
[12:23] sustrik why not
[12:23] sustrik it's up to you
[12:23] pieterh I have a lovely class-oriented model in ZFL...
[12:23] sustrik ack
[12:24] pieterh this would remove the need for random ad-hoc layers like zmsg and s_recv/s_send
[12:24] pieterh ok, I'll think it through
[12:24] sustrik sure, go for it
[12:24] pieterh there's stuff like serialization that other bindings offer, right?
[12:24] pieterh json, etc.
[12:24] sustrik yes, different stuff
[12:24] pieterh hmm
[12:25] sustrik some bindings don't even follow the underlying APIs
[12:25] pieterh indeed
[12:25] sustrik and adhere more to paradigm of the language in question
[12:25] sustrik like C#
[12:25] sustrik Guthur may be more specific on that
[12:26] pieterh this means the real API can be as complex as you need to make it
[12:26] pieterh hmm, ok, this is a good plan
[12:26] sustrik nice
[12:40] Guthur yeah. I'm guilty of deviating somewhat from the underlying API
[12:41] Guthur my rational is that to get accepted by a language community such as C# you need to provide adequate abstractions
[12:41] Guthur string handling being a prime example
[12:42] Guthur they get scared if they see too many byte arrays
[12:43] Guthur I'm not joking there either, when I demonstrated something to my boss that is exactly what he picked up on, he was like 'ugh byte arrays'
[12:43] Guthur I should have hid that better for my demo
[12:45] sustrik :)
[13:15] mile yrashk, got the forst results
[13:15] mile erlang talking to c++
[13:15] mile but not very impressive, I guess I have done something wrong
[13:16] mile got around 2600 msg/sec
[13:16] mile that is, the double of that
[13:16] pieterh mile: are you doing request-reply?
[13:16] mile yes
[13:17] pieterh that's pretty normal then
[13:17] pieterh round-tripping is slow
[13:17] mile ok, thats good in terms I didn't screw up :)
[13:17] pieterh ah, you may also have screwed up, that's not excluded :-)
[13:18] mile hehe
[13:18] pieterh if you want to measure throughput, try a push/pull or pub/sub test
[13:18] mile so, if I get it right, single request/reply are bounded by the roundtrip
[13:18] yrashk there are pub/sub perf tests in erlzmq2
[13:18] mile but if I had parallel clients
[13:18] pieterh yes
[13:18] mile that would scale
[13:19] pieterh mile: how far in the Guide have you read?
[13:19] mile pieterh, good point, I started skipping at some point
[13:19] mile :)
[13:19] mile I'll get back to it
[13:20] pieterh it doesn't talk about round-tripping until chapter 4
[13:20] pieterh but at least you need to know how to scale up req/rep
[13:20] pieterh using devices, which we cover in ch2
[13:21] yrashk btw pieterh I am considering actually using 0mq for some notification bus here :) still not sure whether I can make this simple enough, though..
[13:21] mile that I have read, but I definitely need to read it again :)
[13:21] pieterh yrashk: well, let us know if you hit any blocks
[13:21] yrashk I already hit one, it looks too complicated for me :D
[13:23] pieterh you mean the use case?
[13:23] yrashk yeah
[13:23] yrashk I am still to come to any kind of real model yet
[13:23] pieterh it's a new system or you want to reimplement an existing one?
[13:23] yrashk but I need to build a cluster-wide notification system where everyone can push a mesasge to a "topic" and subscribe to any topic
[13:24] yrashk but it's multihost and host-failure tolerant.. (well, it should be)
[13:24] pieterh read the whaleshark whitepaper yet?
[13:24] yrashk briefly
[13:24] pieterh that's more or less the idea
[13:24] pieterh more or less, not exactly
[13:24] pieterh but there may be ideas there to help you
[13:25] yrashk good idea, thanks
[13:25] pieterh depending on the scale you can also make a brokered model
[13:25] pieterh e.g. something like AMQP's topic exchange
[13:25] yrashk I'd rather avoid amqp
[13:26] pieterh no, not AMQP, just some of the semantics
[13:26] yrashk I mean
[13:26] pieterh I can explain briefly if you like, it's very elegant
[13:26] yrashk I'd rather avoid single-broker thing
[13:26] pieterh ah, ok, why?
[13:26] pieterh 'notification' doesn't immediately suggest high volumes
[13:26] yrashk I need to be very tolerant to host failures (any host can go down any time)
[13:26] pieterh bleh, this is trivial to address
[13:27] pieterh you make live-live redundancy, period
[13:27] pieterh especially simple for pubsub models
[13:27] pieterh so what's the volume?
[13:28] yrashk and yeah there no high volumes -- I just want this system to work through any # of hosts, don't fail when any host is down and make some messages durable
[13:28] yrashk a kind of intra-cluster signalling framework, so to say
[13:28] yrashk serverless one, if you wish
[13:28] yrashk i.e. each host is equal to any other host in the system
[13:28] pieterh ... removing stable points from a network doesn't always make it simpler
[13:29] pieterh atomic multicast?
[13:29] pieterh paxos style?
[13:29] pieterh actually I have an Erlang implementation in my pocket somewhere
[13:29] pieterh but not using 0MQ
[13:29] yrashk I probably don't even care about atomicity. Just getting messages delivered to those who want to listen to them, if the host was down, deliver all durable messages there, too
[13:30] yrashk and yeah, dynamic node additions and removals, obviously
[13:30] pieterh well... IMO you're setting the bar too high
[13:30] yrashk nothing fancy, in fact
[13:30] pieterh yet, too complex to implement... :-)
[13:31] pieterh make it simpler, implement that, then improve it
[13:31] yrashk yep
[13:31] pieterh my suggestion would be to use a broker, and if you need it, a broker pair
[13:31] pieterh use a simple topic routing engine, I can provide you a design if needed
[13:31] pieterh then, when you have that working, figure out what you really want to make
[13:32] yrashk I guess I also need to read about pgm
[13:32] pieterh you can relatively easily evolve from a broker-based architecture to a distributed one
[13:32] pieterh however, jumping straight to a distributed model is rather hard
[13:33] pieterh look at the RRR patterns in Ch4
[13:33] yrashk ikr :)
[13:33] pieterh the distributed pirate is the very last example
[13:33] pieterh pgm is a decent option here but will require network support
[13:34] pieterh you could even make this work with a forwarder device, in a very short time
[13:34] pieterh hah
[13:34] pieterh to cheat...
[13:34] pieterh every node connects to the forwarder front-end, and to the forwarder back-end
[13:35] pieterh use SUB socket filtering
[13:35] pieterh problem solved
[13:35] yrashk so a single forwarded device, then?
[13:35] pieterh yeah
[13:35] yrashk for starters
[13:35] pieterh all it does is connect a SUB to a PUB socket
[13:36] pieterh it acts like a network-global multicast group
[13:36] yrashk yeah I get it
[13:36] pieterh it'll take you about 1 hour to make, maybe less
[13:36] yrashk I can already imagine this, I have some good visual thinking
[13:37] pieterh when it works, send $1.50 to my paypal account, thanks
[13:37] yrashk hahah
[13:37] pieterh :-)
[13:37] yrashk what's the account address? ;))
[13:38] pieterh ah, I have it here somewhere...
[13:38] pieterh so the downside of this approach is that filtering is subscriber-side
[13:39] pieterh and you have a central point, which you can scale to N redundant live-live instances
[13:39] pieterh upside is every node except the forwarder is dynamic
[13:40] yrashk yeah forwarders are obviously the bottlenecks here
[13:41] pieterh it won't slow things down unless you're doing millions of msg/sec
[13:41] pieterh if you need more scale, you switch to a name service approach
[13:41] pieterh i.e. every publisher is also a forwarder
[13:41] yrashk no I am not expecting huge volumes
[13:41] pieterh so, no bottleneck issue
[13:42] pieterh simple code doesn't crash
[13:42] pieterh you probably need to set a HWM properly
[13:43] yrashk so what you're suggesting is having a forwarder on each and every node, right?
[13:44] pieterh no, that's only if you need a lot more scale
[13:44] pieterh forget that
[13:44] pieterh one forwarder in middle of network
[13:44] pieterh binds to a SUB frontend, binds to a PUB backend
[13:44] pieterh every node connects to those two sockets, one to publish, one to subscribe
[13:44] pieterh every node receives every message, and filters using SUB filtering
[13:45] yrashk right
[13:45] pieterh then, if you need it, you double the forwarders and make live/live redundancy
[13:45] yrashk this live-live redundancy, any suggestions on how you would typically do it?
[13:46] yrashk is it like doing the same operations on a # of forwarders?
[13:46] pieterh you run two forwarders, right?
[13:46] pieterh and every node connects its PUB socket to both forwarders, and its SUB socket to both forwarders
[13:46] pieterh then you have to identity each message uniquely (sequence number per publisher)
[13:46] yrashk yeah then I imagined it right
[13:46] pieterh and reject duplicates
[13:46] pieterh 0MQ handles the fanout and fanin properly
[14:50] mikko- hi
[14:51] mikko- I am not quite clear who should prepare the upstream patch for openpgm changes
[14:57] sustrik mikko-_: neither am i
[14:58] sustrik presumably, it's better to patch the master and downstream it afterwards
[15:12] mikko- yes
[15:41] pieterh hi mikko
[15:41] pieterh we can't do pull requests to multiple gits, which is annoying
[15:42] pieterh mikko-_: I'm happy to send a patch from 2.1 to master
[15:45] sustrik pieterh: if you do so, take care to separate the openpgm thing from other functionality
[15:45] pieterh sustrik: yes, it's a single commit, the only grey part is that I tend to update the NEWS at the same time
[15:45] pieterh I'm going to start doing that in a separate commit
[15:45] sustrik ack
[15:46] pieterh I'll send you the patch for OpenPGM then, offlist since it's rather huge
[15:47] sustrik ok, also don't forget to sign it off
[15:47] pieterh right
[15:48] pieterh OK, sent...
[15:49] pieterh there's a better way to do this kind of exchange
[15:49] pieterh but I keep forgetting the ins and outs of git... :-)
[16:08] pieterh sustrik: I'm not going to scream on the list, but I'm using PAIR quite systematically now
[16:23] pieterh ok, I screamt
[16:52] Aaron So I'm trying to build zeromq and have it link against a static uuid library. I'm not very good with configure scripts, and I don't see a parameter to tell configure where the static library and headers are. Can anyone help?
[16:53] pieterh Aaron__: what OS are you on?
[16:54] pieterh and what language are you working in?
[16:54] Aaron Just Ubuntu.
[16:54] Aaron c++
[16:55] pieterh hang on, I'll get you the gcc syntax...
[16:55] Aaron gcc? Don't I need to satisfy the configure check?
[16:55] Aaron first
[16:56] pieterh you are building an application right?
[16:56] pieterh or building zmq?
[16:56] pieterh ah, it doesn't link against anything when you build it
[16:56] Aaron Well, it's a little complicated, but right now I'm just trying to build zeromq.
[16:56] pieterh please use the 2.1 package released today
[16:56] Aaron I am.
[16:56] pieterh ok, good
[16:57] Aaron It builds fine if I use the uuid-dev package.
[16:57] pieterh right, which is what you want
[16:57] Aaron But that creates a dependance on libuuid.so.
[16:57] Aaron Which I don't want.
[16:57] Seta00 Aaron__: should be something like --with-uuid=/path/to/your/custom/build
[16:57] Aaron I have uuid built statically...
[16:58] Aaron ah
[16:58] pieterh Aaron__: when you find out how to do this, add it to the tips page at http://zero.mq/tips
[17:00] Aaron I tried --with-uuid=/path/to/uuid/install/folder but got nothing.
[17:01] Aaron Just the same "configure: error: cannot link with -luuid, install uuid-dev."
[17:02] Aaron I tried a few different directories, /path/to/uuid/install/folder, /path/to/uuid/install/folder/lib, /path/to/uuid/install/folder/include, /path/to/uuid/install/folder/bin, but nothing did anything different.
[17:02] Aaron Does the configure script actually support that argument?
[17:02] pieterh Aaron__: from reading configure, it's looking for, and not finding 'uuid_generate' in -luuid
[17:03] Aaron Hmm, I guess I could double check my uuid.
[17:05] pieterh The rule in configure.in is pretty clear:
[17:05] pieterh AC_CHECK_LIB(uuid, uuid_generate, ,
[17:05] pieterh [AC_MSG_ERROR([cannot link with -luuid, install uuid-dev.])])
[17:06] Aaron Yeah, my uuid library does have a uuid_generate,
[17:07] Aaron Sorry about that.
[17:07] Aaron Barking up the wrong tree.
[17:07] Aaron Thanks for the --with-uuid argument though.
[17:08] Aaron I meant "doesn't have uuid_generate"
[17:10] pieterh ok, problem solved...
[17:14] Seta00 :)
[17:16] Aaron Apparantly there is more than one uuid library out there, sigh.
[17:17] Aaron I thought it was the same one Postgresql was using.
[17:17] mikko- Aaron__: are you building zeromq shared or static?
[17:17] Aaron I want zeromq to be static too.
[17:17] mikko- ./configure --disable-shared
[17:18] Aaron Yeah, I'll pop that on too.
[17:18] mikko- as far as i understand if zeromq is static then libuuid would be as well
[17:18] Aaron Ah
[17:18] Aaron Well, I need to get the right uuid as the first step.
[17:18] mikko- you should be able to use one provided by your OS
[17:19] mikko- usually they ship .a as well
[17:19] Aaron Yeah, I'm working on it as we speak.
[17:20] mikko- on rhel its usually efsprogs and on ubuntu/debian etc uuid-dev
[17:20] mikko- not sure about others
[17:21] mikko- uuid libs are confusing as there are multiple of them
[17:21] mikko- and they are not compatible
[17:24] Aaron Yeah, I was certainly confused.
[17:37] drbobbeaty Question about the 2.1.3 packaging... with the change from 2.1.2 to 2.1.3, I now see that I'm generating a libpgm.so - but I'm not seeing it as an 'ldd' dependency of libzmq.so - is this right?
[17:38] drbobbeaty I used '--with-pgm' in the build, and I see it unpacking the OpenPGM code, and the libpgm.so is generated, but I'm having trouble running my code that worked fine with 2.1.2.
[17:38] Aaron Well, "nm --demangle" tells me I have uuid_generate (0000000000000ee0 T uuid_generate). I'm running configure like so, "../configure --with-pic --prefix=/root/zeromq_install/ --enable-static --disable-shared --without-documentation --with-uuid=/path/to/uuid/install/dir/ ". And I still get "configure: error: cannot link with -luuid, install uuid-dev."
[17:39] drbobbeaty I have tried linking with '-lzmq -lpgm' and still it's not even starting to execute - it's as if I'm missing a critical step.
[17:39] drbobbeaty Anyone run into this?
[17:39] drbobbeaty Pieter?
[17:39] pieterh drbobbeaty: this is indeed a new part of 2.1.3
[17:40] pieterh it's quite possible that the OpenPGM integration doesn't work properly yet :-/
[17:40] drbobbeaty I'm OK with it being new, but code that ran fine with 2.1.2 is core dumping on main() in 2.1.3 - like I'm missing a linking step or something. Is there something I'm missing OTHER than the inclusion of '-lpgm' in the linking step?
[17:41] pieterh drbobbeaty: mikko is the man to answer this
[17:41] drbobbeaty That would be sad for a 'stable' tag.
[17:41] pieterh summon mikko...
[17:41] pieterh drbobbeaty: a bit, yes, but if there's a problem we'll fix it immediately
[17:42] pieterh mikko: are you around?
[17:43] pieterh drbobbeaty: can you post a minimal test case that causes it to die? I'll reproduce it
[17:43] drbobbeaty peiterh: Sure, give me a sec.
[17:43] pieterh k
[18:01] drbobbeaty pieterh: is there a list of what's changed from 2.1.2 to 2.1.3 - I'm having a tough time makeing the "minimal test" as my simple tests are OK, but the biggies that fail are awfully complex. Too complex for a test case, really.
[18:01] drbobbeaty If I knew what changed, maybe it'd help me find the issue in my code.
[18:15] cremes i'm starting to see a new assertion that i haven't run into before
[18:15] cremes Assertion failed: sessions.empty () (socket_base.cpp:127)
[18:15] cremes my code hasn't changed but i did update to the latest master
[18:15] cremes has anyone else seen this?
[18:21] drbobbeaty cremes: not me, but I'm seeing other problems with 2.1.3 that I wasn't seeing in 2.1.2
[18:21] cremes drbobbeaty: what problems are you seeing?
[18:23] drbobbeaty cremes: I'm tracking down an immediate app core dump when using the new 2.1.3 - It's not easily reproducable in a small app, but it's 100% reproduceable in my larger app. I'm trying to trim it down to get a good test case, but it's proving to be hard based on the immediate nature of the core dump.
[18:23] drbobbeaty The small test case runs fine. But the bigger one is nasty.
[18:24] cremes drbobbeaty: if you catch the crash in gdb, where is it crashing inside 0mq?
[18:25] drbobbeaty Nope, it's crashing on main() - the very first executable line. But if I comment out the calls to 0MQ, it runs just fine.
[18:26] drbobbeaty It's the invocation of the ZMQ calls in some way that's causing it to core dump - I just haven't figured out what.
[18:26] cremes that is quite... odd :)
[18:26] drbobbeaty If I had a nice stack trace, I'd be able to figure this out easily. But I don't.
[18:26] pieterh drbobbeaty: are you using pgm in all your cases?
[18:28] pieterh cremes: this is from the development master?
[18:28] cremes pieterh: correct
[18:28] pieterh can you try with the 2.1 release and see if you also get it?
[18:28] cremes i am trying to 'bisect' it now to see if i can narrow down the commit that broke it
[18:28] cremes sure, i can do that
[18:29] drbobbeaty pieterh: Yup, pgm is what I use - 100% of the time.
[18:29] drbobbeaty I'm getting closer... :)
[18:29] pieterh drbobbeaty: ok, so presumably there's a load library issue
[18:30] pieterh the other changes over 2.1.2 are all quite specific patches for core bugs
[18:32] drbobbeaty pieterh: I'm in the configuration code of my app - setting the ZMQ_ properties of the socket. It *appears* I'm getting close.
[18:33] pieterh drbobbeaty: presumably a load error on the openpgm library
[18:33] drbobbeaty In my simple test ( https://gist.github.com/635015 ) I'm now seeing the following error:
[18:34] drbobbeaty Assertion failed: rc == 0 (connect_session.cpp:84)
[18:34] drbobbeaty Double-checking to see if I messed up the test.
[18:38] drbobbeaty pieterh: triple-checked... if I compile this as 'g++ xmit.cpp -o xmit -lzmq' I get the error.
[18:38] drbobbeaty If I add in the '-lpgm' at the end I still get the error.
[18:39] pieterh ok, let me give it a shot
[18:39] drbobbeaty If I switch the order so it reads: '-lpgm -lzmq' I still get the error.
[18:42] pieterh drbobbeaty: I get a clean error message 'Protocol not supported' and then a core dump
[18:42] pieterh this is 0MQ 2.1.3 built without OpenPGM
[18:43] drbobbeaty pieterh: not surprising... I'm using OpenPGM in the example. You need to build it --with-pgm and then try it.
[18:44] drbobbeaty pieterh: I've tried two boxes - Ubuntu 10.04.1 and CentOS 5 and both have the same error when using 2.1.3 w/OpenPGM
[18:44] pieterh drbobbeaty: would you raise an issue at https://github.com/zeromq/zeromq2-1/issues and put a link to your example?
[18:44] drbobbeaty (same gist used on both machines)
[18:45] drbobbeaty I've never done that. I'll try. I'll be as complete as I can be.
[18:45] pieterh interesting... when I build 2.1.3 with pgm, the test program says 'sending...' in a loop
[18:45] pieterh it doesn't crash
[18:45] drbobbeaty Ha... I saw that once, but then when I tried it again, it failed.
[18:45] pieterh ok, will try a few times... this is Ubuntu 10.10 fwiw
[18:46] drbobbeaty The correct behavior is to see the "sending..." in a loop. (until a Ctrl-C)
[18:46] pieterh yes, that's what I get
[18:46] pieterh doing a naive build
[18:46] pieterh so... I'm suspecting you have a conflict between versions of OpenPGM on your systems
[18:47] pieterh or some such mix caused by the upgrade of the OpenPGM version in this package
[18:48] pieterh drbobbeaty: have you enabled core dumps ('ulimit unlimited')?
[18:48] drbobbeaty Yeah, I'm going to check on that now. Thanks for showing that it might be me.
[18:48] pieterh when you get an assertion failure, you need to get the backtrace
[18:52] drbobbeaty pieterh: got the backtrace.
[18:52] drbobbeaty I'll include it if I can't clean up the problem with a drop and load of all the ZMQ libraries on the box.
[18:52] pieterh cool... actually, you can log the issue in the development git, https://github.com/zeromq/zeromq2, it'll be the same
[18:53] pieterh please, do log the assertion and backtrace, it may be useful info
[18:56] sustrik yuck, the change should have gone to master first
[19:01] pieterh sustrik: no-one would have tested it there, and it's not clear that it's caused by a code change
[19:03] pieterh I'd rather see valuable code delivered fast, and fixed fast, than held back...
[19:04] cremes after using git-bisect, it tells me that this is the first bad commit
[19:04] cremes 32ded2b457b2102dba4c15e00363f031d212b1c4
[19:05] cremes Duplicate identities now checked with zmq_connect
[19:05] cremes this commit touched quite a bit of code too
[19:05] cremes pieterh: want me to open an issue against master?
[19:05] pieterh cremes: yes, please
[19:05] cremes i don't have a small repro
[19:06] pieterh it'll be the same issue in master and in 2.1.3
[19:06] sustrik how come it is in stable?
[19:06] sustrik it's an untested patch
[19:06] pieterh sustrik: because we apply all patches to stable automatically
[19:06] pieterh there is no other process for defining what's "tested"
[19:06] pieterh no other way except real use, since we lack comprehensive test coverage
[19:07] sustrik mail from March 16th: "This patch solves issue 176. However, it was not yet tested thoroughly so it should be applied to stable later on, when it's clear it doesn't introduce new bugs."
[19:07] pieterh ah, then that is not in stable
[19:07] pieterh cremes is testing against development master
[19:07] sustrik ok, good
[19:08] pieterh if you tell me to hold back a patch I do so, of course
[19:08] pieterh np
[19:08] cremes https://github.com/zeromq/zeromq2/issues/181
[19:08] sustrik cremes: thx
[19:08] pieterh cremes, just for sanity, can you confirm the problem does _not_ happen on 2.1.3?
[19:08] cremes sure; i will download it and confirm
[19:08] pieterh much appreciated
[19:10] FlaPer87 In case anyone is interested and willing to contribute: https://github.com/FlaPer87/asterisk-zmq-manager
[19:12] mikko pieterh: am now
[19:12] pieterh mikko: possibly a false alarm, we found what might be a problem with OpenPGM in 2.1.3
[19:13] pieterh drbobbeaty is seeing if it's caused by library confusion on his systems
[19:13] pieterh anyhow, pgm example that worked with 2.1.2 fails with 2.1.3... on my system it works fine
[19:14] mikko zfl tests failed as well it seems
[19:14] pieterh FlaPer87: line 412 is lacking 'zmq_msg_close (&message);'
[19:14] mikko hmm, is there a way to test OpenPGM functionally easily?
[19:15] pieterh mikko: fixed that, was some old 2.0 code that magically continued to work on 2.1...
[19:15] mikko i kinda rely on Steven on that as I have no means on testing with any sort of wates
[19:15] pieterh yeah
[19:15] mikko rates*
[19:15] pieterh we have some pgm-capable material at iMatix in Slovakia but it's not configured any longer
[19:16] FlaPer87 pieterh opps, thanks
[19:16] mikko so the openpgm is a false alarm?
[19:16] pieterh FlaPer87: try running under valgrind, you'll catch memory leaks like that
[19:17] pieterh mikko: don't know yet, could be a build issue we've not hit before
[19:17] mikko what OS is drbobbeaty running?
[19:17] FlaPer87 pieterh indeed, I will
[19:17] FlaPer87 thanks :)
[19:17] pieterh Ubuntu and CentOS
[19:17] drbobbeaty mikko: Ubuntu 10.04.1 and CentOS 5
[19:17] mikko drbobbeaty: both fail?
[19:18] drbobbeaty Yup
[19:19] mikko drbobbeaty: it's supposed to link against libtool convenience archive
[19:19] mikko libpgm_noinst.la
[19:19] mikko which just contains a set of object files
[19:20] mikko libpgm.so should not be used when building with zeromq
[19:20] drbobbeaty OK, that's news. In the past (2.1.2), I used the '-lzmq' and it worked just fine with the ZMQ shared lib. Is this not supported in 2.1.3?
[19:21] mikko that should work fine
[19:22] pieterh fwiw I've done 'g++ xmit.cpp -o xmit -lzmq' and the test case runs fine on my Ubuntu box
[19:22] mikko the convenience libtool archive is just a bunch of object files
[19:23] mikko and zeromq uses that rather than the .c files directly
[19:23] mikko in theory (unless there is some libtool weirdness going on) there should be no change how things work
[19:23] Guest31294 in theory, everything always works.
[19:24] cremes pieterh: no assertion when using the 2.1.3 release, so that's a good sign
[19:25] pieterh cremes: thanks for testing that, it means our ad-hoc change flow process is working ... :-)
[19:25] drbobbeaty mikko: I certainly hope that's the case, but up to know I've had a ton of counter-examples staring me in the face. I'll try to clean up things here and see if I can get it back to what works.
[19:25] mikko is there a test code i can run for this openpgm thing?
[19:26] pieterh mikko: https://gist.github.com/635015
[19:28] mikko Initialization complete.
[19:28] mikko sending...
[19:28] mikko sending...
[19:28] mikko sending...
[19:28] mikko and so o n
[19:28] mikko is that the expected result?
[19:29] drbobbeaty mikko: yes.
[19:30] drbobbeaty What I'm getting is the "Assertion failed: rc == 0 (connect_session.cpp:84)"
[19:30] mikko just in case i did rm /path/to/lib/libpgm*
[19:30] mikko and the example still runs
[19:30] mikko can you put nm /path/to/libzmq.so to gist.github.com?
[19:31] mikko and ldd /path/to/libzmq.so
[19:32] pieterh mikko: my guess is that drbobbeaty's two systems have old versions of OpenPGM which is causing a problem
[19:33] pieterh drbobbeaty: any chance you can test on a virgin box? at least one you didn't build on before?
[19:33] drbobbeaty pieterh: I didn't have any old version of OpenPGM as I never had it until ZMQ. In 2.1.2 it was contained within the libzmq.so - with 2.1.3, it appeared that it was external. Now I see it wasn't external at all, just optionally external.
[19:34] pieterh hmm... indeed
[19:34] drbobbeaty pieterh: I'm certainly working on all these ideas. As fast as I can.
[19:34] drbobbeaty I would like to believe it's as simple as a bad version.
[19:34] mikko i would be keen on getting openpgm into daily builds somehow
[19:35] mikko that way we can see if the actual functionality ever breaks
[19:35] pieterh hmm, that would be very nice
[19:37] mikko drbobbeaty: can you paste nm output of libzmq.so ?
[19:42] drbobbeaty Yes, just need a bit. I'm trying to change the RPM packaging to remove the OpenPGM libraries and headers to make it cleaner as to what's really being installed.
[19:46] drbobbeaty mikko: when I run: 'nm /usr/lib64/libzmq.so.1' I get:
[19:46] drbobbeaty nm: /usr/lib64/libzmq.so.1: no symbols
[19:47] drbobbeaty I have cleaned up my RPMs and they are now built as zeromq, zeromq-devel and zeromq-pgm - where the latter has just the OpenPGM libs and headers.
[19:47] drbobbeaty Even after removing everything on my box related to ZMQ, and putting these new RPMs on, I get the assertion.
[19:47] mikko let me install a clean ubuntu VM
[19:47] mikko sec
[19:55] mikko drbobbeaty: i removed my installation of libzmq completely, rebuilt
[19:55] mikko rebuilt the xmit.cpp and ran with LD_LIBRARY_PATH
[19:55] mikko it seems to work ok
[19:56] mikko i'll make an install-exec-hook patch later so that people dont get confused with libpgm.so
[19:56] mikko i dont think there is a way to force autotools not to install it
[19:56] mikko brb
[19:56] drbobbeaty mikko: Thanks for the checking... I'm getting convinced it's now I have it built, but it's not at all clear why the way I build it from 2.1.2 to 2.1.3 causes these issues. I've had many upgrades in this process.
[19:58] mikko drbobbeaty: i will install clean ubuntu vm later and test
[19:58] mikko it's worrying if this actually fails for anyone
[19:59] pieterh drbobbeaty: can you post an issue with the assert and backtrace?
[19:59] drbobbeaty mikko: I'm troubled by what I'm seeing, and I fear it's in the building. I'm going to see if I can build it without the RPM structure if it works. I'm guessing that's how you guys are all doing it. Just 'configure --with-pgm' - right?
[19:59] pieterh it may be possible to figure out what's happening from that
[19:59] pieterh drbobbeaty: yes
[20:00] drbobbeaty pieterh: OK, trying that now
[20:00] pieterh just as explained in the docs; configure; make; sudo make install; sudo ldconfig
[20:00] pieterh --with-pgm, then
[20:00] mikko drbobbeaty: yes
[20:01] mikko drbobbeaty: it's possible that the build breaks if for example RPM does something very weird
[20:01] mikko i think install-exec-hook should allow cleaning the libpgm stuff
[20:01] drbobbeaty mikko: it's the same spec file that ships with the package.
[20:02] mikko drbobbeaty: i know, i don't think it's very actively checked (it seems)
[20:02] mikko i had RPM builds running earlier but it was in a chroot which was really slow
[20:02] mikko if there are actually RPM users i could install centos VM for rpm packaging
[20:02] pieterh drbobbeaty: I see at least one problem with the spec file in 2.1.3
[20:02] mikko so that the packaging gets tested regurly as well
[20:03] pieterh it *might* be causing corrupted RPMs...
[20:03] mikko now i need to run for a min ->
[20:06] pieterh drbobbeaty: if you want a fixed spec file, https://gist.github.com/880103 should be better
[20:06] pieterh I doubt this could cause the test program to crash but you never know
[20:06] flexo hey. as i didnt find an answer to my question in either the guide, reference or faq (well, actually the lack of an answer might be an answer...) here it goes: does zmq support unreliable transport? (pure udp unicast/multicast)
[20:06] pieterh flexo_: nope
[20:07] flexo any plans to add support for it?
[20:07] pieterh flexo_: feel free to send us a patch...
[20:07] flexo :)
[20:07] pieterh :-)
[20:08] drbobbeaty pieterh: what changed? Something small, or a lot of things?
[20:08] pieterh drbobbeaty: in the spec file, removal of the stand-alone device programs
[20:08] pieterh in the core library, OpenPGM integration and a lot of small patches
[20:08] pieterh it's in the NEWS and ChangeLog
[20:08] pieterh except the spec file, which I forgot to fix until just now
[20:09] flexo seriously, would it fit into zmq's design.. purpose.. whatever? (looks to me like quite some work was done to _guarantee_ that the messages arrive...)
[20:09] pieterh flexo_: it could be useful for certain apps, yes
[20:09] flexo for example in the app i have in mind :)
[20:10] pieterh yeah, when real time performance is more important than reliability
[20:10] flexo yep
[20:10] pieterh seriously, if you need it, and if you can write C++, it should be doable
[20:10] flexo but there are also certain kinds of messages which must arrive intact, hence having something like zmq for that kind of stuff would be nice
[20:10] pieterh transports are pluggable in 0MQ
[20:11] pieterh you can use different sockets for different QoS
[20:11] flexo pieterh: assuming that i am a c++ wizard, how long would think it would take me to do it?
[20:11] flexo (i know zmq for 10 minutes :)
[20:11] pieterh if you also know UDP, 1 day to get it working, 2-3 more days to get it working real nice
[20:12] flexo alright
[20:12] pieterh you start by asking the question on the list, "I intend to add UDP support, please tell me what files to create / modify"
[20:12] flexo im pretty confident i can figure it out by myself...
[20:13] pieterh we'll help package it all (docs, etc.) afterwards
[20:13] pieterh well, take a look at pgm integration as an example
[20:13] pieterh actually I should point out that it's technically impossible
[20:13] flexo uhm?
[20:13] pieterh no-one has ever done this before, make 0MQ work with UDP
[20:13] pieterh therefore it's impossible
[20:13] pieterh and perhaps even illegal
[20:14] flexo i see :(
[20:14] pieterh :-)
[20:14] flexo just fyi: the app i have in mind would be a flight simulation visual
[20:14] pieterh flexo_: I'm kidding, you'd be the first person to try it
[20:14] flexo i.e. several render boxes which receive telemetry via multicast
[20:15] pieterh yes, excellent use case
[20:15] flexo (which must get the data really fast. ;)
[20:15] pieterh but doesn't mind if there's some breakage
[20:15] drbobbeaty mikko: why are you passing in '--prefix=/usr/local' to the OpenPGM configure script within the main configure script? Isn't that assuming that I'm putting it there? I'm not.
[20:15] flexo (and otoh there is some stuff which must arrive, although it doesnt matter if the latency is somewhat higher, i.e. you change the scenario, daytime, etc.)
[20:16] flexo add a new entity into the simulation, that kind of stuff
[20:16] pieterh yeah, it's like VoIP
[20:16] pieterh mix of reliable and unreliable patterns
[20:16] pieterh UDP would make a great addition
[20:17] pieterh but impossible and perhaps even illegal, so don't tell anyone...
[20:17] flexo i was about to make up some protocol using a mix of tcp and multicast udp myself.. but i suppose it might be less work to just add unreliable transport support to zmq
[20:17] flexo sure
[20:17] pieterh well, advantage of layering it under zmq is you get access to nice semantics on top
[20:17] flexo its even intended for military usage so its morally nasty too
[20:17] pieterh plus APIs in lots of languages
[20:17] pieterh morally nasty, I love it
[20:18] pieterh I mean, terrible, terrible stuff...
[20:18] flexo yes, and the zmq API looks really nice
[20:18] flexo and well-looking APIs are rare :)
[20:18] pieterh it's going to get polished some more for v3.0
[20:19] flexo a far-fetching thought here... is there some project providing some highlevel RPC mechanisms on-top of zmq? (think CORBA)
[20:21] pieterh flexo_: see the Guide, Chapter 4
[20:21] pieterh there are a whole set of RPC-like patterns built on top of 0MQ
[20:22] pieterh plus sample code and protocol specs
[20:22] pieterh not CORBA, though
[20:22] pieterh rather more 2011
[20:23] flexo chapter 4? reliable request-reply?
[20:23] pieterh yes
[20:23] flexo i realize that you can do all that stuff on-top of zmq
[20:23] flexo but an ORB actually _does_ provide some pretty nice stuff
[20:24] pieterh specifically...?
[20:24] pieterh not that I'm disagreeing, just curious to what you mean
[20:25] flexo well, the basic thing would be the IDL compiler - more advanced stuff would be object keys, activation / incarnation, etc.
[20:26] flexo its pretty easy to do distributed garbage collected servants
[20:26] pieterh i prefer garbage collecting servants
[20:26] pieterh actually have no idea what those terms mean
[20:26] flexo well
[20:27] flexo take a virtual filesystem for example
[20:27] pieterh ok
[20:27] flexo built using interfaces like IStream, IFile, etc.
[20:27] pieterh sure
[20:27] flexo you can connect to any component in our application, no matter where it runs and ask it to open a file for you
[20:27] pieterh sure
[20:28] flexo it provides transparent encryption / decryption, there is a key ring manager which asks the user for passwords, etc.
[20:28] pieterh this can all be layered on top of 0MQ
[20:28] flexo the ORB provides me with the means to implement something like this in a nice object-oriented manner without writing lots of the low-level management code
[20:29] flexo so i just say "give me a POA with this policy", i put the filename into the object key, my servant manager gets invoked when someone wants to invoke a ::read() on the the stream
[20:29] mikko drbobbeaty: autotools probably pass down prefix automatically
[20:30] pieterh POA = piece of action? sorry, the jargon is not familiar to me
[20:30] pieterh so why are no companies making money from ORBs any more?
[20:31] flexo portable object adapter.. an internal CORBA component which. hm.. is responsible for servants. it knows their servant manager and how to invoke methods on these servants
[20:31] flexo heh
[20:31] flexo pieterh: dont get me wrong here
[20:31] flexo CORBA heavily sucks in a lot of areas
[20:31] pieterh nah, I'm just curious, CORBA has always appeared to me as massively complex
[20:31] pieterh complex concepts, complex APIs, complex tools
[20:31] flexo but most of its "complex" design actually makes sense
[20:32] pieterh sure, but that doesn't mean much in the end
[20:32] flexo actually
[20:32] pieterh sadly, a 100% complete solution that is 100x more complex than an 80% complete solution will not win
[20:32] flexo well
[20:32] flexo you definitly have a point there
[20:32] pieterh this is what I mean by 2011
[20:32] flexo my company made money with an ORB i wrote couple of months ago
[20:33] flexo but this ORB only implemented the features needed, so maybe it was like 70% conformant :)
[20:33] pieterh if you have customers for ORBs, you may enjoy using 0MQ as the transport
[20:33] flexo (targeting embedded hardware)
[20:33] flexo yea, im thinking about that
[20:33] flexo im also thinking about replacing CORBA because it annoys me heavily...
[20:33] pieterh that's how we see 0MQ, as a smart transport layer
[20:34] flexo its very cumbersome to use, especially with the c++ mapping
[20:34] pieterh it's never going to include IDLs and key managers
[20:34] flexo do you think IDL compilers are bad in principle?
[20:35] pieterh no, just that data representation isn't the job of the messaging fabric
[20:35] flexo (i.e. the whole RPC idea which makes it look like a function/method call)
[20:35] pieterh for most purposes, JSON is fine
[20:35] flexo i completely agree
[20:35] pieterh JSON is very 2011
[20:35] flexo heh
[20:35] pieterh totally cheap and nasty serialization that works for 80% of developers
[20:35] flexo too nasty for my purposes
[20:36] flexo the parsing overhead just...
[20:36] flexo no
[20:36] flexo it feels wrong.
[20:36] pieterh so you ideally want a JSON approach for low-volume cases, and smart binary framing for high-volume cases
[20:36] pieterh I'm also a fan of divide and conquer
[20:36] flexo i dont really see the advantage of JSON over CDR
[20:36] flexo CDR is very nice IMHO
[20:36] pieterh JSON requires no upfront thinking
[20:37] flexo neighter does CDR if you are used to think of a struct like a damn struct in memory :)
[20:37] pieterh so why is no-one using CDR?
[20:37] pieterh I mean, the market decides
[20:37] flexo pieterh: because CDR is deep within CORBA
[20:37] flexo and CORBA scares people
[20:37] pieterh nah, because it's a complex toolchain
[20:37] flexo no, CDR is complety separete of everything
[20:38] flexo its simply a binary data representation
[20:38] flexo containing some stupid padding.. but thats far less worse than JSON
[20:38] pieterh I can't even find a web page describing CDR
[20:38] flexo as i said
[20:38] pieterh it's embedded in some PDF somewhere
[20:38] pieterh fail
[20:39] flexo thats the point.. its embedded deep inside the CORBA specs
[20:39] flexo and thats the reason no-one uses it
[20:39] flexo not because its bad
[20:39] pieterh yeah, but that's not "very nice"
[20:39] flexo agreed
[20:39] bpl29 Hello hello. I am trying to link the zmq library to my project, but think I have built an 1386 library instead of x86_64.
[20:39] flexo but "everyone uses it" doesnt make it a nice format.. take bittorrent for example
[20:39] bpl29 *i386
[20:39] flexo everyone uses bittorrent and bittorrent uses a highly stupid serialization format
[20:40] pieterh flexo_: well, I've made dozens, hundreds of technically brilliant tools over the years
[20:40] pieterh all '
[20:40] pieterh 'perfect'
[20:40] pieterh all dead, because they were inaccessible
[20:40] flexo its nice to parse with some regular expressions, but if you try to make a very-high throughput implementation you turn crazy because of that "human readable" ascii mess
[20:40] pieterh bpl29: have you copied the library from one box to another?
[20:41] flexo pieterh: well. i dont disagree with you on any particular point...
[20:41] pieterh anyhow, wrt ORB capabilities over 0MQ, go for it, I'd say
[20:41] bpl29 pieterh: No. As well, I just rebuilt from source. I am linking with -L/usr/local/lib -lzmq
[20:42] flexo i just refuse to acknowledge that i spend so much time working with crap
[20:42] pieterh flexo_: Sturgeon's Law
[20:42] flexo yea :)
[20:42] bpl29 pieterh: The error simply states "warning: in /usr/local/lib/libzmq.dylib, file was built for unsupported file format which is not the architecture being linked (i386)"
[20:42] pieterh bpl29: so why do you think you built it wrong? it should work automatically, your compiler knows it's on a 64-bit box
[20:42] pieterh hmm
[20:42] pieterh what OS are you on?
[20:43] bpl29 pieterh: OS X
[20:43] pieterh bpl29: sorry, I'm not expert in OS X builds, but maybe someone else here can help.. cremes?
[20:43] cremes pieterh: what's up?
[20:44] pieterh any idea why a build on OS X would say "file was built for unsupported file format which is not the architecture being linked (i386)"
[20:45] cremes yes...
[20:45] cremes the library is being built i386 but it's linking against a (probably) x86_64 binary
[20:46] cremes bpl29: run "file <name>" against the library and tell us the output
[20:46] bpl29 okay one moment
[20:46] pieterh "If you're using Xcode, then by default Debug mode only builds your platforms native arch. So on a new mac, this would be an x86_64 binary."
[20:46] yrashk it looks like I've got a case of send() lockup with ZMQ_NOBLOCK :-\
[20:46] pieterh http://www.qtcentre.org/threads/30670-On-Mac-quot-...file-was-built-for-unsupported-file-format..-quot-SOLVED
[20:46] pieterh yrashk: cool!
[20:46] flexo pieterh: anyway, just wanted to clarify that it seems to me that you are one of the more.. much more.. competent people i have met. although i disagree with your JSON notion. :)
[20:47] bpl29 creme: /usr/local/lib/libzmq.dylib: Mach-O 64-bit dynamically linked shared library x86_64
[20:47] pieterh flexo_: I'm just old and cynical, not particularly competent, but the compliment feels nice, thanks
[20:47] cremes bpl29: and what program is trying to load that library? run "file" against it too
[20:47] yrashk pieterh: indeed :)
[20:47] flexo pieterh: well, i am highly cynical aswell.. maybe thats the reason for the sympathy :)
[20:47] flexo (not old though. :D)
[20:48] yrashk I think I know why it happens
[20:48] pieterh flexo_: one thing I've learned is that intelligence is a product of many minds, not a few experts
[20:48] pieterh yrashk: 2.1.3 or development master?
[20:48] yrashk but no idea how to fix it
[20:48] pieterh what do you think happens?
[20:48] yrashk 184bdb8e2bb6fe252af1bf4dd16d1c9ed8892f7a
[20:49] yrashk with this specific command that I am sending to another thread, that thread closes the socket that its being sent through
[20:49] bpl29 creme: I am using ocamlmklib.
[20:49] yrashk so sometimes it gets through fine (it closes the socket *after* send() is over)
[20:49] bpl29 owait, sry, misinterpreted
[20:49] jond flexo_ CDR is an ok encoding, though the character set stuff is dreadful. havent used corba for years though
[20:49] yrashk but sometimes it closes it while send() is still working
[20:49] yrashk and send() locks
[20:49] yrashk even though it's in NOBLOCK
[20:49] yrashk or should I just upgrade to dev master?
[20:50] flexo jond: i agree that it could be improved in some areas... i am just saying i prefer it over json :)
[20:50] jond flexo have you seen BSON?
[20:50] pieterh yrashk: that commit is in development master
[20:50] yrashk but are you aware of this issue I described? or whether it is an issue at all?
[20:51] pieterh not aware of the issue
[20:51] pieterh you distracted me by posting a long UUID
[20:52] pieterh yrashk: if you can log an issue, that'd be good, it sounds like an edge case
[20:52] flexo jond: looking at it right now... what immediatly caught my attention... no unsigned integers?
[20:52] yrashk I am still experimenting
[20:52] bpl29 creme: I'm not quite sure where that binary is. Do you think it could be i386? I can search more.
[20:52] jond flexo: always thought it would be nice to rip the structs / types part of idl out to use as message format generator, leaving all the interface stuff behind, idl looks so much better than an xml or some schema
[20:52] bpl29 creme: ocamlmklib, that is
[20:52] flexo jond: oh yea.
[20:53] flexo especially if the IDL happens to look almost exactly the same in the language you are developing in... :)
[20:54] flexo (but even if it doesnt fit... i like our python testcases, using omniorbpy)
[20:54] flexo there are some nice really nice things about CORBA, the mean problem is that its one huge "solution", not a set of solutions layered on-top of each other
[20:54] cremes bpl29: i'm not really sure what you are doing
[20:54] jond flexo : yep, but remember that python corba binding in omniorb almost makes corba bearable, so much easier than the c++
[20:54] flexo *main
[20:55] flexo jond: true...
[20:55] cremes bpl29: are you running the examples using the ocaml bindings?
[20:55] flexo (the ICE mappings look like nice by the way)
[20:55] bpl29 cremes: I am writing new OCaml bindings.
[20:56] cremes bpl29: ok; how do you execute your ocaml runtime?
[20:56] flexo sigh. but project management will never let me deviate from the "industry standard" CORBA.... :)
[20:57] jond flexo: well henning was pretty damning in the end and thought he could better, even thought CDR was no better than XDR and the Zero Ice would be the way to go. Last time I looked Vinoski had discovered erlang
[20:57] bpl29 cremes: the command in question is "ocamlmklib -o _wrapper_stubs wrapper.o -L/usr/local/lib -lzmq"
[20:57] flexo hehe
[20:58] cremes bpl29: ok; type "which ocamlmklib" and it will tell you where that program is in your PATH
[20:58] cremes then run "file" on it
[20:58] bpl29 cremes: thanks. I knew it was something so simple.
[20:58] jond flexo: where I am we use XDR and our own message defn language generating all the marshall/unmarshall classes. No corba
[20:59] flexo jond: yea.. could have been done for our project too... but as i said.. "industry standard" :)
[21:00] flexo (the department i am working for still has some 200mhz SGI boxes where they actually develop for)
[21:00] bpl29 cremes: "/usr/local/bin/ocamlmktop: POSIX shell script text executable"
[21:00] bpl29 cremes: I think that's not what I'm looking for
[21:00] cremes bpl29: umm... you are going to have to do some digging
[21:00] bpl29 cremes: the bash script just calls ocamlc, so I'll go file that
[21:00] cremes that script must be calling some binary that is trying to link a 32-bit binary against the 64-bit 0mq lib
[21:01] jond flexo: that's corporate life....
[21:01] flexo yep
[21:01] flexo OTOH this is the reason i know some MIPS assembly :)
[21:01] cremes bpl29: you might need to pass some flags when linking your ocaml code so that it makes everything 64-bit
[21:01] bpl29 cremes: Okay, thanks Cremes. I think I can do something with that. You've been a big help to me, and you also helped me yesterday, so thank you very much.
[21:01] yrashk still can reproduce on master
[21:01] flexo (i maintain a very small library which runs on said boxes... had to debug it there once :)
[21:01] yrashk hmmm
[21:02] cremes bpl29: ok; at this point, you'll need to jump into the ocaml channel and ask them for some help :)
[21:02] yrashk but only on my slower laptop, not mac pro
[21:02] yrashk heh
[21:02] bpl29 cremes: That's a good idea. I'll do that then. Thanks again.
[21:02] flexo the dept. is getting into trouble because its getting harder and harder to get replacement parts for these boxes :)
[21:02] flexo the funny thing about the harddisks is that if the power goes out and they cool down about 1/3 of them dies
[21:03] flexo so the boxes must be online all the time
[21:03] flexo and i really enjoy taking interns into that room
[21:05] flexo they dont quite believe it otherwise :)
[21:06] flexo the number of people who know how to administer them grows shorter every year too
[21:06] cremes i just added one of my wish-list items to the 0mq 3.0 roadmap wiki page
[21:06] flexo sheesh. where am i working :)
[21:07] pieterh cremes: a page is like... 60 lines of 80 chars?
[21:08] cremes pieterh: a page is usually some multiple of 512 bytes, like 4k on linux
[21:09] pieterh so, almost 60 x 80 like I thought :-)
[21:09] cremes almost
[21:09] cremes heh
[21:10] pieterh nice change request, but you may have to submit the patch yourself
[21:10] cremes pieterh: understood
[21:31] Guthur hehe, I love how CORBA was defined by OMG
[21:32] Guthur I have this strange feeling it is very fitting]
[21:50] mikko drbobbeaty: i got a patch that disables shared library building on openpgm side
[21:51] mikko i will chat with steven about it and see what he things
[21:51] mikko thinks*
[21:52] drbobbeaty mikko: thanks, but I found the problem. You didn't see it because etho0 is OK for you, but on my Ubuntu box, they bonded the NICs so I could use eth0, I had to use bond0.
[21:52] drbobbeaty tricky admins.
[21:53] pieterh drbobbeaty: heh, and this worked with 2.1.2?
[21:53] drbobbeaty It worked when I HAD 2.1.2 - they did it in the last week.
[21:54] pieterh drbobbeaty, ah, just retesting 2.1.2 would have saved us all some time
[21:55] drbobbeaty Yeah, that was my fault, and I'm really sorry about that. It just didn't cross my mind because it was working.
[21:56] pieterh glad pgm is working, anyhow
[22:00] mikko drbobbeaty, pieterh good!
[22:01] mikko in any case i worked up solutions on zeromq and libpgm side to prevent the confusing libpgm.so install
[22:01] mikko pieterh: you are CCd on mail to steven
[22:01] mikko let's see what he thinks
[22:01] pieterh mikko: reading it now...
[22:01] drbobbeaty Thanks, mikko... I think it'll be a very good thing to have.
[22:02] pieterh mikko: I'd guess not installing the libpgm.so is best
[22:04] mikko yeah
[22:04] mikko would be the best if steven agrees with the patch
[22:06] mikko productive day
[22:07] pieterh yeah, I've almost finished the clone pattern
[22:07] pieterh *and* I took the kids to the park
[22:07] pieterh great day
[22:11] Guthur would exposing the connected IPs to a 0MQ be possible
[22:11] pieterh Guthur: doubt it
[22:13] Guthur shame, I was just thinking in terms of a FIX engine, one of the recommend authentication procedures was to match IPs to companies
[22:13] Guthur obviously spoofing is possible, which the whitepaper also mentions
[22:13] pieterh it's been discussed before
[22:13] Guthur oh
[22:14] pieterh one might have a pluggable authentication layer that sits above the transport
[22:14] pieterh this would be necessary for any public-facing 0MQ server IMO
[22:14] pieterh though people have argued stuff like banning bogus clients is a firewall issue
[22:15] Guthur security is always should a head pickle
[22:16] pieterh indeed... :-)
[22:16] Guthur Better I at least making a passing effort to understand the options now though
[22:17] pieterh if you take 0MQ today, rather than some possible future version
[22:17] pieterh then validation must be done explicitly using authentication
[22:18] pieterh alternatively you can make a TCP front-end that speaks FIX to the world, and 0MQ to the internal network
[22:18] Guthur that was a feature I would like
[22:18] Guthur I was going to build the 0MQ version first though
[22:18] pieterh it's a fairly simple app, if you have FIX libraries
[22:18] Guthur then add the bridge to TCP later
[22:19] Guthur I was going to take QuickFIXs message parsing for now
[22:19] Guthur and replace the engine with 0MQ
[22:19] pieterh if you asked me, I'd make the bridge first
[22:20] pieterh use QuickFIX parsing over TCP at one side, 0MQ at the other side
[22:23] Guthur probably the more sensible approach
[22:23] Guthur I was just really eager to build a big 0MQ system, hehe
[22:28] pieterh btw, an open source FIX-to-0MQ bridge would be very useful for Whaleshark
[22:32] Guthur the question in my mind is how much functionality to put in the bridge
[22:33] pieterh filtering and extraction
[22:33] pieterh so that you chop the message flow down as far as possible
[22:33] pieterh without doing too much CPU intensive work
[22:34] pieterh IMHO
[22:34] Guthur well the easy one is splitting the admin and app message
[22:34] Guthur messages*#
[22:35] Guthur would a bridge offer a place to add encryption/decryption
[22:36] pieterh probably not, unless you can offload that work
[22:37] Guthur so add the basic level possibly filter admin/app messages, and verify checksums
[22:37] Guthur add/at
[22:37] pieterh it depends really on the volumes you're dealing with and your hardware
[22:37] pieterh you want to get the machines working hard enough but not stressed
[22:38] pieterh otherwise you're going to get latency spikes when there's a lot of data
[22:38] Guthur FIX engines can be running on some pretty hefty hardware
[22:38] Guthur the bank I work for is now serving up all their FX pricing on a FIX engine
[22:39] pieterh you work for a bank...
[22:39] Guthur and now even clients are asking for it to be exposed them
[22:39] Guthur pieterh, yep
[22:39] Guthur lol
[22:39] Guthur if only I was writing the cheques
[22:40] Guthur in truth though this is a personal project
[22:40] pieterh an open source FIX-to-0MQ bridge would be neat
[22:41] Guthur yep, it's my goal to have this stuff open source
[22:41] Guthur if I can build it properly
[22:41] mikko pieterh: sent you some stuff again
[22:41] pieterh make it open source first, then you'll be sure to build it properly
[22:42] mikko don't read it before morning, it's work related
[22:42] mikko build the community first, good products will follow
[22:42] mikko said some mozilla guy
[22:42] pieterh mikko: I'm supposed to *sleep* knowing there's an unread mail from you?
[22:42] pieterh unpossible
[22:43] mikko heh, im gonna sleep now. another long day ahead
[22:43] mikko night
[23:00] pieterh anyone here using 0MQ from C?
[23:07] Guthur when something does not work I now use it for a sanity check
[23:07] Guthur but i don't think that really counts
[23:18] pieterh I've sketched out a proposal for a high-level C binding
[23:18] pieterh http://www.zeromq.org/bindings:c
[23:21] pieterh well, g'nite everyone...
[23:32] Guthur g'night pieterh

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy