IRC Log

Thursday January 20, 2011

[Time] NameMessage
[00:52] cremes andrewvc: what's up?
[00:57] cremes andrewvc: just send me an email...
[01:43] mvolkov I am connecting to multiple sockets with "tcp://" in attempt to loadbalance load. Result: only first socket does the work, the rest do fail with following error: "Assertion failed: msg_->flags & ZMQ_MSG_MORE (rep.cpp:80)"
[01:45] mvolkov I am using pyzmq. zmq.PUSH --->zmq.PULL
[01:45] mvolkov is that a bug?
[04:21] StockMQ Hi All!
[04:26] StockMQ I am developing a middleware which will basically listen feed from Exchange, decompress it, read and process it and then publish the cleansed feed to subscribers
[04:27] StockMQ after reading the elaborate docs and articles.. I have arrived to the decision to use 0MQ.. but have a few queires
[04:27] StockMQ The first one being.. the feed from the exchange is a UDP multicast
[04:28] StockMQ so in my middleware app can i listen to it using PGM SUB
[04:44] cremes StockMQ: that's a good question for the mailing list; there is more pgm expertise there than on irc usually
[04:47] guido_g in principle you need to convert between the formats, ømq itself has its own wire format
[04:49] StockMQ ohk.. I was looking to know that.. can i connect my zeroMQ client to a non zeroMQ server
[04:49] guido_g as i said, ømq has its own wire format
[04:49] StockMQ the reason being i understand that zeroMQ is based on messaging and frames
[04:50] guido_g just use the normal way to connect to your fix/fast stream and read the data as usual
[04:51] guido_g then you might send it out using ømq
[04:54] StockMQ guido_g:
[04:54] StockMQ by normal way you mean
[04:55] guido_g the you did it before
[04:55] guido_g *the way
[04:55] StockMQ ohk
[04:56] StockMQ So it is better i connect to the fast stream using my VC++ application
[04:56] StockMQ Then maybe i can implement a pipeline in 0MQ where the worker threads can process the feed packets and then send it downstream to Publisher
[04:57] StockMQ Does that sound right
[06:06] StockMQ How different is Push/Pull from Publish/Subscribe??
[06:07] StockMQ Apart from the fact that Subscribe makes filtering easy
[06:09] guido_g see http://api.zeromq.org/zmq_socket.html
[06:53] CIA-21 zeromq2: 03Dhammika Pathirana 07master * rc91bf25 10/ (src/decoder.hpp src/zmq_engine.cpp):
[06:53] CIA-21 zeromq2: Fix handle connection reset during session init
[06:53] CIA-21 zeromq2: Patch to handle nmap version probes.
[06:53] CIA-21 zeromq2: Signed-off-by: Dhammika Pathirana <dhammika@gmail.com> - http://bit.ly/hZescC
[07:07] Evet would you use zeromq as front-end for several protocols? for example, an http server
[07:12] guido_g what do you mean w/ frontend?
[07:16] Evet guido_g: im planning to build my web app in c; i think zeromq would be a good framework for request/respond based tcp servers
[07:17] guido_g no its not
[07:17] guido_g ømq is not a tcp lib
[07:17] guido_g it uses its own wire format
[07:27] Evet guido_g: then, which event loop library you suggest for bsd sockets?
[07:28] guido_g the one you have most knowledge of
[09:27] mikko good morning
[09:45] Evet howdy mikko
[11:07] kabs Hello, I was going through the guide on ZeroMQ at http://zguide.zeromq.org/chapter:all#toc0, If you use c code directly from it under the heading "Getting the Message Out", which is a simple pub-sub code, it doesn't work.
[11:08] kabs Sever is subscribing data but client is stuck at this step sscanf (string, "%d %d %d", &zipcode, &temperature, &relhumidity);
[11:09] kabs Anyone , who can tell me how I can debug this client code, or does this code work, I found the code above it to be working , now sure about this pub-sub code, please help ...
[11:33] sustrik are you sure the client is able to connect to the server?
[13:10] stockMQ Hi. Just started with 0MQ today
[13:10] stockMQ am using it with VC++
[13:10] stockMQ I am trying to create the socket in constructor
[13:10] stockMQ use it in other method of my class
[13:12] stockMQ But the constructor socket_t (const socket_t&); seems to be private in zmq.hpp
[13:12] mikko stockMQ: isnt that copy constructor?
[13:14] sustrik it is
[13:15] sustrik you cannot copy the socket
[13:15] mikko this->socket = new socket_t(context, ZMQ_PUSH);
[13:15] mikko and in destructor delete it
[13:15] sustrik or:
[13:15] mikko initializer list is another option i guess
[13:15] sustrik socket_t s (context, ZMQ_PUSH);
[13:20] stockMQ ok
[13:20] stockMQ lemme try the above
[13:28] stockMQ this->publisher = new socket_t(context, ZMQ_PUSH);
[13:28] stockMQ gave
[13:29] stockMQ error C2512: 'zmq::socket_t' : no appropriate default constructor available
[13:34] sustrik strange
[13:35] sustrik can you have a look at your zmq.hpp file?
[13:35] sustrik there should be this line there:
[13:35] sustrik inline socket_t (context_t &context_, int type_)
[13:35] sustrik is it?
[13:35] stockMQ yes
[13:35] mikko and what is 'context' in this case?
[13:35] stockMQ i see that line
[13:35] sustrik zmq:::context_t&
[13:36] mikko i mean in this 13:28 < stockMQ> this->publisher = new socket_t(context, ZMQ_PUSH);
[13:36] sustrik zmq::context_t ctx(1);
[13:36] sustrik zmq::socket_t s(ctx, ZMQ_PUSH);
[13:36] mikko if type of context was something funny it wouldnt find constructor i assume
[13:36] stockMQ context_t context(1); socket_t publisher(context, ZMQ_PUSH);
[13:36] stockMQ This work without any problem
[13:37] sustrik great
[13:37] stockMQ My issue is i need a pointer to publisher socket in other method of my class.
[13:37] mikko i see
[13:37] stockMQ so if i do
[13:37] stockMQ socket_t publisher(context, ZMQ_PUSH);
[13:37] mikko the member needs to be zms::socket_t *publisher;
[13:37] stockMQ the visibility is restricted to the constructor
[13:37] mikko not zmq::socket_t publisher;
[13:38] mikko as the latter would try to call empty constructor (assuming no initializer list)
[13:40] sustrik you can either make the socket member of the class, or you can pass it *pointer* to the socket as an argument
[13:40] stockMQ that makes sense
[13:40] stockMQ but i am referring to https://github.com/imatix/zguide/blob/master/examples/C++/durapub.cpp
[13:40] mikko stockMQ: not sure i understand
[13:42] stockMQ Ok here is the scenario
[13:43] stockMQ 1. I receive fast feed from the exchange which i want to inturn publish to other subscribers
[13:44] stockMQ 2. So i want to intialize the publisher in the constructor
[13:44] stockMQ and Send whenever a packet is recvd from the fast feed
[13:44] stockMQ sustrik...did u see that link
[13:44] sustrik yup
[13:44] sustrik why not make the socket a member of the class?
[13:45] stockMQ yes i want to do that
[13:45] stockMQ for example
[13:45] stockMQ socket_t* publisher;
[13:45] stockMQ In constructor
[13:45] stockMQ publisher = ? ? ?
[13:46] sustrik new socket_t (ctx, ZMQ_PUSH);
[13:46] mikko context must be a member or passed as param as well
[13:49] mikko so for example https://gist.github.com/01386aebde16bfb65672 should work
[13:51] mikko sustrik_: i took a brief look at UDT sockets
[13:52] stockMQ Thanks guyz
[13:52] sustrik mikko: what have you found out?
[13:53] sustrik stockMQ: np
[13:53] mikko sustrik_: 1. there are two libraries, neither of them packaged with distros it seems
[13:53] mikko they got their own type for sockets
[13:53] mikko zeromq seems to be keen on "fd_t" in places
[13:54] sustrik you mean, user-space pseudosockets, right?
[13:54] mikko yes
[13:54] sustrik ok
[13:54] stockMQ I am new to the world of pointers.. Just curious.. zmq::socket_t publisher (context, ZMQ_PUB); publisher.bind("tcp://*:5565");
[13:54] stockMQ how would this work
[13:55] stockMQ should it not be publisher->bind
[13:55] mikko stockMQ: no, because publisher is not a pointer
[13:55] sustrik stockMQ: if you are not familiar with pointers I would suggest using C API
[13:55] sustrik it's less complex
[13:55] mikko sustrik_: however their api is pretty close to normal socket api
[13:55] stockMQ ok
[13:55] sustrik mikko: i see
[13:56] mikko sustrik_: http://udt.sourceforge.net/udt4/index.htm
[13:56] mikko if you look at the example code it probably looks familiar immediately
[13:56] sustrik yes, it does
[13:58] sustrik there looks there is no way to get the underlying file descriptor
[13:58] mikko yes, it's very likely that you can't
[13:59] sustrik that would mean that each UDT worker would have to run in a separate thread
[13:59] mikko they seem to have their own epoll implementation
[13:59] mikko UDT::epoll
[14:00] sustrik the problem is that it accepts only UDT sockets
[14:00] sustrik so it can't be combined with TCP
[14:00] sustrik or PGM
[14:00] sustrik or whatever
[14:00] mikko yes
[14:00] mikko and probably we wouldn't like to use theirs as main implementation
[14:01] sustrik Ah!
[14:01] sustrik lrfds
[14:01] sustrik [out] Optional pointer to a set of system sockets that are ready to read.
[14:01] sustrik lwfds
[14:01] sustrik [out] Optional pointer to a set of system sockets that are ready to write, or are broken.
[14:01] sustrik so there's a way to combine UDT sockets with OS sockets
[14:02] sustrik we would have to create a new poller though
[14:02] sustrik one that uses UDT epoll
[14:02] sustrik instead of select/poll/epoll/kqueue etc.
[14:03] sustrik it's doable though
[14:04] mikko luckily it should be very similar to epoll
[14:05] sustrik yeah, i think it can be done
[14:05] sustrik the question is whether anyone needs that kind of thing
[14:06] sustrik (UDT via 0MQ i mean)
[14:07] mikko the other option would be to run separate thread?
[14:07] mikko oh
[14:07] mikko got you now
[14:07] mikko yes, it might be slightly faster than tcp assuming there is no network issues
[14:08] mikko not sure if anyone has burning need for that sort of speeds
[14:08] sustrik it would be nice to have, the obvious question is who will bear the maintenance burden
[14:08] sustrik and whether it's worth of it
[14:09] sustrik viable approach would be to cooperate with the author of UDT library
[14:09] sustrik same way as we do with steven
[14:10] mikko that would be ideal i guess
[14:10] mikko the other udt library is by bittorrent
[14:10] mikko someone mentioned a github link
[14:11] mikko hmm, it's actually utp
[14:11] mikko https://github.com/bittorrent/libutp
[14:12] mikko that sounds a bit like UDT
[14:14] sustrik we had few people asking for UDP-based transports
[14:14] sustrik however, afaik we have no idea what specifically they are interested in
[14:14] sustrik it may be:
[14:15] sustrik 1. unreliable transport without re-transmissions
[14:15] sustrik 2. UDT-like fast transfer for large chunks of data
[14:15] sustrik 3. unreliable multicast
[14:16] sustrik it would be good figuring that out...
[14:16] mikko there was one guy here the other night
[14:16] mikko he was after 1.
[14:16] mikko but i dont think udp alone fits into zeromq model that well
[14:17] sustrik you can at least do pub/sub
[14:18] sustrik anyway, there are many different transports that can be implemented
[14:18] sustrik sctp, udp+dccp, udt etc.
[14:18] sustrik what's missing imo is a clear interface for writing new transports
[14:18] mikko it looks like there is no strict abstraction of transports at the moment
[14:18] sustrik exactly
[14:18] mikko from the experience of starting udt impl
[14:20] sustrik that's something that i would like to have, however, it's not clear how the interface should look like
[14:22] sustrik mikko: any idea?
[14:22] mikko it's very hard to say
[14:22] mikko there are so many parts to the interface
[14:22] mikko such as receiving / sending, polling etc
[14:22] sustrik :(
[14:23] mikko and some sockets might have their own poll, some might have callback driven interface
[14:23] sustrik yuck
[14:23] mikko it's very hard to create a "plug&play" api for all these
[14:23] sustrik well
[14:23] sustrik we have an interface for different polling mechanisms
[14:24] sustrik there's already select, poll, epoll, kqueue & dev/poll
[14:24] mikko because thinking about the larger picture you probably want pluggable transports that users can create
[14:24] mikko which would drive contributions
[14:24] sustrik yes
[14:25] sustrik so, i think the polling part is not that big a problem
[14:25] sustrik so for example
[14:25] mikko true
[14:25] sustrik with udt you would write a udt_epoll polling mechanism
[14:25] sustrik and...
[14:25] sustrik a sec
[14:26] sustrik add following to the poller.hpp:
[14:26] sustrik #elif defined ZMQ_HAVE_UDT
[14:26] sustrik typedef udt_epoll_t poller_t;
[14:26] mikko yeah
[14:27] sustrik hm, one problem would be if there are 2 such transports, the pollers would collide
[14:28] mikko or maybe we limit to transports that have pollable fd
[14:28] mikko but does that limit too much?
[14:29] mikko maybe it would be possible to have pollable fd in udt but they havent had the request yet
[14:29] sustrik i don't think so
[14:29] sustrik thay have to use sockets underneath anyway
[14:29] mikko yes
[14:29] sustrik so it's only a question of exposing them to the user
[14:29] sustrik which is nasty
[14:29] mikko if the fd was strictly for polling
[14:29] sustrik but needed every now and then
[14:30] sustrik yes
[14:30] mikko integration is prime example
[14:30] mikko people using udt only wont probably need it
[14:30] sustrik same with 0mq itself, there's ZMQ_FD
[14:30] sustrik OpenPGM does the same thing
[14:31] mikko if udt exposed an fd it would be far from tcp socket
[14:31] mikko wouldnt*
[14:31] sustrik right, that's the another question
[14:32] sustrik how to abstract the socket behaviour
[14:32] sustrik i.e. how should the transport interface dealing with actual connection negotiation and transfer look like
[14:33] sustrik my feeling is that there are 2 distinct cases:
[14:33] sustrik 1. connected transport
[14:33] sustrik 2. unconnected transports
[14:33] sustrik example of 1. is TCP, example of 2. is PGM
[14:33] mikko yes
[14:33] mikko common things for both are:
[14:33] mikko read/write, connect/bind
[14:34] mikko some sort of "is alive"
[14:35] sustrik actually, connect/bind doesn't make much sense for 2.
[14:35] sustrik with pgm, both connect and bind do the same thing
[14:35] sustrik which would be more properly called "initialisation"
[14:35] mikko yes, probably initialization
[14:35] mikko hmm
[14:36] mikko i assume getting pollable fd would be a common thing
[14:37] sustrik probably
[14:38] mikko ideally the transport would be as dummy as possible
[14:38] sustrik the transport would provide an fd, 0mq would use it *only* for polling
[14:38] mikko shifting bytes left and right
[14:38] sustrik and forward the handling of the event back to the transport
[14:38] mikko yes
[14:39] sustrik sorry?
[14:39] sustrik whay bytes?
[14:39] sustrik what
[14:39] sustrik ah, the messages, right?
[14:39] mikko yes
[14:39] mikko but from transport point of view bytes
[14:40] mikko just data
[14:40] sustrik hm, that would limit the range of possible transports severely
[14:40] sustrik think of HTTP transport
[14:40] sustrik it has to know about message boundaries
[14:41] sustrik so that it can attach HTTP header to each message
[14:41] mikko i don't think http transport is realistic in core
[14:41] sustrik it's just an example
[14:41] sustrik think of SCTP
[14:41] sustrik it has its own framing
[14:41] sustrik thus it needs to know about 0mq message boundaries
[14:41] sustrik so that it can frame each 0mq message into SCTP message
[14:42] stockMQ sorry to interrupt guys
[14:42] stockMQ using this->context = new zmq::context_t (1); this->publisher = new zmq::socket_t (*this->context, ZMQ_PUB); //publisher(context, ZMQ_PUB); this->publisher->bind("tcp://*:5565");
[14:42] mikko sustrik_: hmm
[14:43] stockMQ and whenever i receive a packet from exchange which is in ms
[14:43] mikko sustrik_: let's think about it in this way:
[14:43] mikko i am sending 1MB message
[14:43] sustrik stockMQ: what's wrong with that?
[14:43] stockMQ i do
[14:43] stockMQ s_send (this->publisher,"TEST");
[14:43] stockMQ here the code breaks with an Access Violation
[14:43] mikko you probably want s_send(*this->publisher, "TEST");
[14:43] stockMQ but instead if on every packet receipt
[14:43] stockMQ if i do
[14:44] mikko stockMQ: how is s_send defined?
[14:44] stockMQ context_t context(1); socket_t publisher(context, ZMQ_PUB); //publisher(context, ZMQ_PUB); publisher.bind("tcp://*:5565"); // s_send() s_send (publisher,"END");
[14:44] sustrik i think s_send works on C API
[14:44] sustrik not C++ API
[14:44] stockMQ it works fine
[14:45] mikko stockMQ: did you try s_send (*this->publisher, "TEST"); ?
[14:45] mikko sustrik_: so
[14:45] mikko so what i am thinking is for example sending large message which cant be sent in one go
[14:45] mikko would the chunking be handled in transport?
[14:47] sustrik can't be sent in one go = multi-part?
[14:58] mikko no
[14:58] mikko transport layer
[14:58] mikko EAGAIN for example
[16:28] Skaag weird, i'm compiling on a fresh machine a very simple test and I get test.c:(.text+0x3c): undefined reference to `zmq_socket', etc.
[16:44] mikko Skaag: are you linking against zmq?
[16:44] mikko Skaag: looks like not
[17:28] sustrik mikko: re
[17:28] sustrik i see
[17:29] sustrik in any case 0mq is not able to send a message that doesn't fit into memory
[17:29] mikko sure
[17:29] mikko i mean if you have non-blocking socket
[17:29] sustrik and how do large messages affect whether trasport should frame messages or not?
[17:29] mikko you send 200K, get EAGAIN
[17:29] mikko does the transport buffer the message and retry?
[17:30] sustrik you mane EAGAIN from the underlying socket?
[17:30] mikko yes
[17:30] sustrik TCP socket
[17:30] sustrik the transports are event driven
[17:31] sustrik so once the underlying tcp socket is ready for writing
[17:31] mikko ok
[17:31] sustrik transport is notified
[17:31] sustrik and tries to write some data
[17:31] sustrik how much is up to the transport
[17:31] sustrik anyway, the data can be sent partially
[17:32] sustrik so the transport has to remember the reamining part
[17:32] sustrik and send it on next "can send" event
[17:32] mikko so the buffering of messages happen all the way down in transport?
[17:33] sustrik transport has to buffer one message
[17:33] sustrik (the partially sent one)
[17:33] mikko ok
[17:33] sustrik the remaining messages are buffered in pipes
[17:33] mikko so the write () method for transport would take a message
[17:34] sustrik it flow of command is in opposite direction
[17:34] sustrik it's not that pipe calls write() function on transport
[17:34] sustrik rather trasport calls read() on the pipe
[17:35] mikko hmm
[17:35] sustrik it's poller who actually invokes the transport
[17:35] sustrik saying "you can read from TCP socket"
[17:35] sustrik or "you can write to TCP socket"
[17:47] mikko sustrik_: im gonna make windows dlls available
[17:47] mikko from daily builds
[17:47] mikko as "snapshots"
[17:47] sustrik mikko: great
[17:49] sustrik mingw or msvc?
[17:50] mikko both even
[17:51] lt_schmidt_jr about jzmq maven package is making eclipse quite unhappy because of the file hierarchy
[17:51] lt_schmidt_jr I have fixed it up locally
[17:51] sustrik then submit a patch to jzmq project
[17:51] mikko jzmq doesnt use maven?
[17:51] lt_schmidt_jr there is a pom.xml
[17:52] lt_schmidt_jr Mikko is that a question ?
[17:52] mikko it was a question in a way
[17:52] mikko i've only used the autoconf build
[17:52] lt_schmidt_jr and the jar?
[17:53] mikko oh, there is a pom
[17:53] lt_schmidt_jr sans instructions I used the autoconf build and then ran mvn install
[17:54] lt_schmidt_jr seemed to do the right things - but the file hierarchy was a bit off
[17:54] lt_schmidt_jr eclipse would compile but complained bitterly and would red the dependencies
[17:54] lt_schmidt_jr it was not wrong - just not quite regular
[17:55] lt_schmidt_jr sorry - this is a new thing for me - how do I submit a patch?
[17:56] sustrik send it to the mailing list
[17:56] lt_schmidt_jr as a tar.gz?
[17:56] sustrik ah
[17:56] lt_schmidt_jr ok, will do
[17:56] sustrik a diff presumably
[17:57] lt_schmidt_jr ok
[17:57] lt_schmidt_jr since I have you here: I was looking at this thread
[17:58] lt_schmidt_jr http://lists.zeromq.org/pipermail/zeromq-dev/2010-July/004411.html
[17:59] lt_schmidt_jr is this a serviceable way to create a local message bus ?
[17:59] sustrik what do you mean by "serviceable" ?
[17:59] lt_schmidt_jr will this work?
[17:59] lt_schmidt_jr :)
[17:59] sustrik yes
[17:59] sustrik also check the devices
[18:00] sustrik the component in the middle comes precompiled with 0mq
[18:00] sustrik zmq_forwarder
[18:00] lt_schmidt_jr I was assuming I could use the ZMQForwarder on a Java thread
[18:00] mikko for jzmq you could possibly send a pull request?
[18:00] mikko or do they follow the same model of signed patches?
[18:01] sustrik i don't think so
[18:01] sustrik lt_schidt_jr: sure, you can do that
[18:02] lt_schmidt_jr thank you - and I should be able to have sockets connecting to the device specify what they are interested in - the thread some insist on subscribing for "all"?
[18:02] sustrik you can subscribe for whatever you want
[18:03] lt_schmidt_jr sustrik_: excellent, thank you
[18:03] sustrik you are welcome
[18:23] sustrik normal sockets = TCP sockets?
[18:23] sustrik if so, then the normal socket would have to speak 0mq wire protocol
[18:23] sustrik see here rfc.zeromq.org
[19:49] mikko sustrik_: http://snapshot.valokuva.org/
[19:49] mikko now each build generates a snapshot
[19:50] sustrik great
[19:50] sustrik is that mingw or msvc build?
[19:50] mikko msvc
[19:50] mikko i think i need to add that notion
[19:50] sustrik i see
[19:51] sustrik maybe link it from the website somewhere...
[19:51] mikko mingw doesnt pass make check on windows at the moment
[19:51] mikko due to ipc lacking
[19:52] mikko maybe ipc tests should be excluded on mingw
[20:03] sustrik ipc is missing on win altogether
[20:40] mikko sustrik_: sent two patches to list
[21:02] sustrik ok, let me see
[21:07] sustrik mikko: what's the visibility stuff about?
[21:13] CIA-21 zeromq2: 03Mikko Koppanen 07master * r8561a55 10/ src/zmq.cpp :
[21:13] CIA-21 zeromq2: Remove unnecessary visibility pragmas
[21:13] CIA-21 zeromq2: Signed-off-by: Mikko Koppanen <mkoppanen@php.net> - http://bit.ly/haVGtq
[21:14] CIA-21 zeromq2: 03Mikko Koppanen 07master * r8e61a11 10/ tests/Makefile.am :
[21:14] CIA-21 zeromq2: Do not execute ipc tests under MinGW
[21:14] CIA-21 zeromq2: Signed-off-by: Mikko Koppanen <mkoppanen@php.net> - http://bit.ly/e2t6TF
[21:15] sustrik ok, applied
[21:15] sustrik luckily, i am not a maintainer of the build system, i don't have to understand what's going on :)
[22:08] mikko sustrik_: the visibility is defined in declaration
[22:08] mikko as far as i understand
[22:08] mikko so the implementation doesn't really need that pragma
[22:08] mikko and we fixed a bug regarding visibility where we moved from that pragma push/pop to ZMQ_EXPORT
[22:09] mikko so it seems that statement had no effect
[23:07] Skaag mikko: yes, sorry, the -lzmq line was commented out in the makefile for some weird reason...

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy