Tuesday July 19, 2011

[Time] NameMessage
[01:29] rev whoever set up this irc client on the zmq webpage - thanks a lot! my office won't let me install an irc client so this is great.
[01:55] michelp rev, glad you could join us :)
[13:46] ptrb any problem to re-close an already closed socket?
[13:47] mikko why does it need to be re-closed?
[13:47] sustrik sure, it's a problem
[13:47] sustrik the socket is deallocated during first close
[13:48] sustrik second close accesses non-existent object
[13:48] ptrb I see. Does C++ destructor trigger a close()?
[13:48] sustrik yes
[13:48] ptrb I see.
[13:49] sustrik however, in C++ binding destructor tests whether socket was closed explicitly before
[13:49] sustrik and if so, it doesn do zmq_close()
[13:49] ptrb So, assuming RAII ownership model of a socket, I should never clo... oh, alright.
[13:50] ptrb Very good! Thanks.
[14:27] amirpc Hey guys I'm putting together an architecture document for a proposal at work and one thing I'm looking to put in there is current production use cases, to try to head of dissenters before they can raise the issue - does anyone know where I can find some good examples?
[14:28] mikko amirpc_: you have read the guide i assume?
[14:28] mikko one company i know using zeromq is loggly
[14:28] mikko they speak quite openly about it
[14:28] amirpc mikko: Yep, I have read the guide but I read code samples, etc a long time ago - is there prod info in it? gonna go check
[14:29] mikko what sort of info are you looking for?
[14:29] mikko just people using zeromq in production?
[14:30] amirpc Yeah, if I don't have examples then the change fearers will rapidly take the conversation in a bad direction
[14:30] amirpc just other people I can say who are using it
[14:33] mikko you can probbaly spot such a companies from mailing-lists
[14:34] amirpc okay I'll give that a shot
[14:34] taotetek amirpc_: we (aggregate knowledge) use zeromq in production
[14:34] taotetek amirpc_: we have a few blog posts up about it on our corporate blog if that's helpful
[14:35] taotetek amirpc_:
[14:36] taotetek amirpc_: we use rsyslog as an "artery" to steam log data to services that then distribute log data to various parsing and analaysis systems via zeromq
[14:36] taotetek s/steam/stream
[14:37] pieterh amirpc_: there are a number of 'stories' on the site:
[14:37] pieterh there could be more, we've not been very good about collecting them
[14:37] taotetek oh hi pieterh :)
[14:37] pieterh hi Brian!
[14:38] pieterh ohai
[14:38] taotetek pieterh: just sitting down to work on some prototypes and play more with jruby + zeromq
[14:38] pieterh saw that, nice...
[14:38] taotetek pieterh: heading out to HQ in california in a couple of weeks trying to get some good examples together for the dev team
[14:39] pieterh how many devs?
[14:40] amirpc pieterh: Do you know what our largest scale deployment is? They'll ask about that, we've got a few million users a day and this will be spread across prob 200 individual nodes, ever seen anything that big?
[14:40] taotetek pieterh: we've got hmmm.. just 4 engineers in the dev group, well plus the cto does a lot of coding and I do a lot of coding
[14:40] taotetek pieterh: so I guess 6 or so of us writing code on a regular basis
[14:41] pieterh amirpc_: no idea what the largest scale 0MQ deployment is, but I know for a fact some people are aiming at 10M+node networks
[14:41] pieterh taotetek: have you read
[14:41] amirpc okay if I wanted to find more info about that, mailing list is best or?
[14:41] taotetek pieterh: oh and ken writes code of course when he's contracting for us
[14:41] pieterh amirpc_: ask on zeromq-dev... but many large deployments are discrete about their use of 0MQ
[14:42] pieterh taotetek: sounds fun... :)
[14:42] taotetek pieterh: yup! it was a nice article
[14:42] amirpc okay well I just really want to use zeromq in my project but I'm going up against a decent number of people who are afraid of change, so any advice you can give would be helpful :)
[14:42] pieterh i've used this approach a few times now, it works well to go from hello world to real prototypes
[14:43] pieterh amirpc_: if you are fighting emotional arguments (fear), don't come with technical answers
[14:43] taotetek pieterh: we used a version of "urban protocol" for our first workflow system speaking of unprotocols ;)
[14:44] pieterh taotetek: :-) nice
[14:44] taotetek pieterh: experimenting with majordomo now
[14:44] amirpc lol I appreciate that, but it's from technical people so they fear-monger using antiquated examples of how change has introduced instaiblities before
[14:44] pieterh I did this with a team, wrote a protocol and we then made stacks
[14:44] pieterh and then we switched to a OHAI form for the protocol and they loved it
[14:45] pieterh amirpc_: there are some _very_ large firms starting to use 0MQ
[14:45] taotetek pieterh: yup. it made people laugh and got some "ICANHAZ?" jokes flying around for a few days
[14:45] taotetek amirpc_: we don't have a "wide" system (not a lot of nodes), but we run a lot of data through it
[14:45] pieterh amirpc_: the truth is that technical change is always risky
[14:46] pieterh there is no point in arguing otherwise, accept it, work with it
[14:46] taotetek amirpc_: 500 to 700 million "events" a day
[14:46] pieterh so to reduce the risk of change, you need a number of strategies
[14:46] pieterh such as rapid prototyping, portability of OS and language, and wide, supportive community of users
[14:47] taotetek amirpc_: I'll tell you this - it hasn't been as entirely smooth as we'd hoped (what ever is), and we did run into a surprise here or there in the library that threw us a curve ball or two
[14:47] taotetek amirpc_: but we've not yet ever regretted the decision to go with zeromq as a core technology
[14:47] pieterh amirpc_: essentially, a message-based architecture is extremely good at reducing risk
[14:48] amirpc taotetek: Are you allowed to say what company it is, and/or what curve balls you've seen?
[14:48] taotetek amirpc_: that's us
[14:48] pieterh amirpc_: where are you based?
[14:49] amirpc SF
[14:49] taotetek amirpc_: one of the curves was just conceptual. it took a little bit to fully grok that zeromq is not a messaging solution so much as it is a toolkit to enable you to build your solutions
[14:50] pieterh You might want to contact dotcloud, they're quite large 0MQ users in SF
[14:50] pieterh they built their cloud architecture using AMQP/RabbitMQ and then moved to 0MQ
[14:50] taotetek amirpc_: the second was that the "high water mark" settings don't quite do entirely what we first thought
[14:50] pieterh taotetek: if you need it, I have a good little credit-based flow control model working
[14:51] taotetek amirpc_: e.g. with push / pull pipeline you will still end up with quite a few messages in transit in the buffers no matter what you set the HWM at
[14:51] pieterh avoids the need for HWMs on router-based architectures
[14:51] taotetek pieterh: I'd be interested in seeing it
[14:51] taotetek pieterh: the latest things we're working on are using REQ / REP and just allowing the workers to ask for more jobs when they want them
[14:51] taotetek pieterh: avoids routing, avoids HWM issues, etc
[14:52] amirpc What I want to use it for is less for the queue and more for the durability
[14:52] pieterh taotetek: yes, Majordomo
[14:52] amirpc and flexibility to allow novel architectures to arrise
[14:52] pieterh amirpc_: you do need to read the Guide and internalize the "language"
[14:53] taotetek pieterh: yup, the next step we're taking is majordomo - allowing workers to register the services they can handle, etc
[14:53] amirpc I've read the guide and implemented several projects with zeromq already actually
[14:53] taotetek pieterh: our first production system was a bunch of task workers that all perform the exact same task so it was a good, simple place to start.
[14:53] taotetek pieterh: now that people feel comfortable with that we'll evolve
[14:54] pieterh taotetek: I'm posting that CBFC example now, with some explanation
[14:54] pieterh amirpc_: if you get to the stage where you need some evangelism / explanation, I'm sometimes in SFO, and happy to travel there
[14:55] taotetek pieterh: excellent, looking forward to reading it
[14:56] amirpc pieterh: That's good to hear. I probably will either sink or swim today @ 5pm cst, hehe. If they don't get hung up on zeromq, this project has 10s of millions of dollars behind it and staggering reach
[14:59] chrismurf Hello all.
[15:00] pieterh taotetek:
[15:02] pieterh chrismurf: hi
[15:02] taotetek excellent, thanks pieterh
[15:03] pieterh it requires bidirectional sockets, so router-dealer is the usual choice
[15:03] chrismurf Are there architectural obstacles to implementing a *non-reliable* multicast protocol in zeroMQ? I would like the option to use UDP Multicast for some messages, and wonder if that's something that can be done /in/ 0mq.
[15:03] pieterh with the new generic sockets in 0MQ/3.x, this will work even better
[15:03] sustrik chrismurf: no, there's no real problem
[15:04] pieterh chrismurf: I have a UDP framework but it's rather complex still -
[15:04] pieterh sustrik: lol
[15:04] chrismurf sustrik, we have been using LCM ( which is UDPM based
[15:04] sustrik is that UDP multicast
[15:04] chrismurf yes
[15:04] sustrik pieter: i though it was UDP unicast
[15:05] pieterh sustrik: that's UDP unicast but it's not that different
[15:05] chrismurf considering switching to 0mq and google protobuf, but I'm not sure of all the consequences
[15:05] pieterh it already for example does broadcast connects (i.e. connect to any IP address)
[15:05] sustrik chrismurf: you can take src/pgm_sender.hpp/.cpp and src/pgm_receiver.hpp/.cpp as a template
[15:05] sustrik remove the PGM stuff and replace it by UDP
[15:05] sustrik should be easy
[15:05] chrismurf sustrik, thanks - perfect.
[15:06] fintler chrismurf: if you plan on doing a ton of rpc, consider thrift
[15:06] chrismurf fintler, I'll look at it, but I'm not RPC-heavy
[15:06] chrismurf mostly just data sharing
[15:07] chrismurf IPC on an autonomous underwater robot
[15:07] fintler proto buffers is probably better for just data
[15:09] chrismurf pieterh, sustrik, fintler, thanks!
[15:09] pieterh chrismurf: what kind of traffic volume are you doing?
[15:09] chrismurf well, the problem is that some of the network cables go through wet-mateable cables
[15:10] chrismurf which can have HORRIBLE SNR
[15:10] chrismurf and we have gigabit ethernet based cameras, trying to go through a 100Mb link, through wet-mated connectors, toss in a little seawater...
[15:11] chrismurf steady state traffic isn't huge
[15:11] pieterh do you need pubsub, or is point-to-point OK?
[15:12] pieterh so look at the vtx_test_udp.c source in that project I pointed to
[15:12] chrismurf it's a research vehicle, so we're trying to switch to pubsub, so that it is easy to write standalone applications that can consume data without running in a critical process
[15:12] pieterh it demonstrates all the 0MQ socket types over UDP
[15:12] pieterh including pub-sub, which is lossy
[15:12] pieterh request-reply will do basic reliability
[15:13] pieterh it should be simple to plug into your code:
[15:13] pieterh #include "vtx.c"
[15:13] pieterh #include "vtx_udp.c"
[15:13] chrismurf pieterh, thanks
[15:13] pieterh it is really not meant for public use yet, but it handles nicely in tests
[15:13] pieterh also it's not fast, does blocking writes
[15:14] pieterh however it effectively gives you 0MQ sockets connected via UDP in all the different patterns
[15:14] chrismurf okay
[15:15] chrismurf I have been happy with the way LCM handles UDPM (ie - invisibly) so I may also look at how complex their implementation is (both are LGPL)
[15:16] pieterh I guess the benefit of using 0MQ would be you get a single API and framework for all different kinds of transport
[15:16] pieterh UDP to your seawater cameras, TCP to other boxes on the network, inproc to other threads in the process
[15:17] chrismurf pieterh, sort of, yeah - we're looking at using TCP for some critical messaging (ABORT!) but generally udpm/pubsub for message passing so things remain modular
[15:18] chrismurf we have a mass-o-threads right now, which is impossible to test, debug, and develop for
[15:18] chrismurf and message passing is a mix of UDP and TCP and shared memory
[15:18] pieterh chrismurf: 0MQ makes that really simple
[15:18] chrismurf so tracking data flow is impossible
[15:18] pieterh indeed
[15:19] chrismurf so we want to switch to a bunch of separate processes, with pubsub between them.
[15:19] chrismurf but udpm falls down nicely as traffic goes up, and I don't know enough about pgm
[15:19] chrismurf and there are scary sounding caveats about loopback interfaces and the like on the 0mq website ;-)
[15:20] chrismurf I've found we start running code on one pc, and then it's on two, and then it's on one again, etc... udpm pub/sub means we don't have to care.
[15:21] chrismurf why are there no unreliable transports in 0mq now? Just lack of need by the developers?
[20:59] mikko pieterh: here?
[20:59] pieterh mikko: hi!
[20:59] mikko looks like 2-1 and 2-2 builds on win are failing
[20:59] pieterh just writing my book
[20:59] mikko not sure if the builds are wrong or a new patch
[21:00] pieterh I saw that, not sure how that's possible, seems int64_t isn't defined...
[21:00] mikko ermm, 2-2 only
[21:00] pieterh czmq or zeromq2-2?
[21:00] mikko mailbox.obj : error LNK2001: unresolved external symbol "public: void __thiscall zmq::signaler_t::recv(void)" (?recv@signaler_t@zmq@@QAEXXZ)
[21:00] mikko
[21:01] mikko
[21:01] mikko looks like one of those backports broke it
[21:02] pieterh hmm...
[21:02] pieterh LINK : warning LNK4098: defaultlib 'MSVCRTD' conflicts with use of other libs; use /NODEFAULTLIB:library
[21:02] mikko thats fine
[21:03] mikko openpgm debug build
[21:03] pieterh normally there are the same backports to 2.1 as 2.2
[21:03] mikko
[21:03] mikko it happens with mingw32 as well
[21:03] mikko signaler.cpp: In member function 'void zmq::signaler_t::send()':
[21:03] mikko signaler.cpp:123:54: error: invalid conversion from 'unsigned char*' to 'const char*'
[21:03] pieterh let me check that...
[21:04] mikko another one is signaler.cpp: In member function 'void zmq::signaler_t::recv()':
[21:04] mikko signaler.cpp:197:54: error: invalid conversion from 'unsigned char*' to 'char*'
[21:05] pieterh ok, so I'd expect it to fail on 2.1 as well
[21:05] mikko let me check 2.1
[21:05] pieterh same source on 2.1 and 2.2 for this file
[21:05] mikko
[21:05] mikko same fail
[21:06] pieterh ok, I see the problem, code was fixed in 3.x but not in the back port
[21:06] pieterh 30 seconds...
[21:07] rev mikko: are you working on
[21:08] mikko rev_: nope
[21:08] pieterh mikko: that should do it now
[21:08] pieterh thanks for catching this...
[21:08] mikko rev_: i've done occasional patches for jenkins
[21:09] rev mikko: ok i saw 'hudson' in the build output and just wondered.
[21:09] mikko rev_: it's a jenkins build cluster for zeromq
[21:09] mikko runs a few operating systems and compilers
[21:09] mikko
[21:10] rev mikko: gotcha
[21:21] devinus what message format would you guys use for RPC?
[21:24] mikko devinus: depends
[21:24] mikko devinus: i was just reading this
[21:26] devinus msgpack was something i had in mind
[21:26] devinus right now we're doing jsonrpc over http
[21:27] devinus thinking of using zeromq for the service
[21:34] whack I've found msgpack quite nice doing rpc in C
[21:34] whack only because the msgpack library makes it easy to write data structures; I could just as soon use xdr but fewer languages support that