IRC Log

Tuesday October 11, 2011

[Time] NameMessage
[00:22] Guest25624 hi there. would someone be kind enough to explain why this incredibly simple snippet stops working the second go-around of the infinite loop? http://paste.pocoo.org/show/490784/
[00:22] Guest25624 once the loop iterates, I get zmq.core.error.ZMQError: Operation cannot be accomplished in current state.
[00:43] Guest25624 ah, well I refactored that code anyhow. Turns out I really didn't need multiple sockets at all.
[00:46] mikko Guest25624: that is usually EFSM error
[00:46] mikko for example if request socket has sent a request it must receive reply next
[00:46] mikko Guest25624: in your case it just "recv, recv, recv"
[02:06] przemeklach I'm building an application using Java on Windows. I build the msvc project and I have my all but I'm not quite sure how to use the .dll in java. Can someone help? Thanks in advance.
[02:06] przemeklach sry I mean .dll not all
[09:03] sustrik mikko: hi
[09:09] sustrik mikko: build.zero.mq seems to be dead
[09:54] mkoppanen heh, a few days installing the whole build stuff everything things go down
[09:54] mkoppanen -things
[09:54] mkoppanen whoa, so many mistakes in one sentence
[09:54] mkoppanen sustrik: there?
[10:08] sustrik mkoppanen: hi
[10:33] da-loic hello, short question: pear.zero.mq appears to be down, is it a known issue?
[10:36] sustrik build.zero.mq is down as well
[10:36] sustrik i guess it's the same issue
[10:45] da-loic thanks Martin, so it will be solved in due time (building from source instead)
[10:50] sustrik yes, it's mikko's (mkoppanen) hardware, he's aware of the failure
[10:50] da-loic sustrik, thanks for the details!
[13:06] nicolas hello
[13:11] geekstuff hi. is the pear.zero.mq server down ?
[13:15] nicolas if memcpy() is used on every send, then I am required to always send POD types?
[13:19] drbobbeaty nicolas_: yes, but that's not horrible… you can always take your objects and place them into a (char *) array or even an STL vector, etc. You would have to handle the serialization and deserialization anyway.
[13:19] drbobbeaty ZMQ doesn't do the serialization/deserialization for you.
[13:22] nicolas I see
[13:22] nicolas i'm using protocol buffers for some tasks
[13:23] nicolas but for the performance critical ones i'm using just c structs
[13:24] drbobbeaty That makes perfect sense… PBs is nice… and structs are OK too, but then you need to have a way to memcpy() the structs into a linear memory buffer, and send that. Then on the other end, get the data and memcpy it back to a struct.
[13:24] drbobbeaty Pointers don't survive, so you have to be careful there. But I'm sure you have that all figured out.
[13:24] nicolas yes.. I wrote a templated static method to send()
[13:24] nicolas but it works only with PODs
[13:25] nicolas recv() too
[13:25] drbobbeaty That's the advantage with protocol buffers… it "costs" time, but it can allow you to forget about the serialization and deserialization.
[13:26] nicolas I wrote a Send<MyPod>(socket, my_pod)
[13:26] nicolas don't know if it is a good design practice, but it works fine
[13:27] drbobbeaty Until you see a problem, it's good enough.
[13:28] nicolas Do you recommend any other serialization libs out there, besides PB ?
[13:29] drbobbeaty MessagePack - at http://msgpack.org/
[13:30] drbobbeaty I have used PB for a while and have seen where it fails, and while I haven't used MessagePack, I'm looking at it for the next time I need something like this as it appears to get around the problems I've seen with PB and that's a big plus to me.
[13:31] nicolas like what problems with PB?
[13:31] nicolas MessagePack looks good
[13:32] drbobbeaty The biggest are when you have to change versions. It doesn't handle multiple versions of PB on the same transmission scheme very well. If you're only using one version, it's fine. But that wasn't the case. It was a long-standing install, and as a consequence, it had several versions of PB, and they always caused problems. We had to standardize on a version of PB, and that sort-of defeats the purpose of the whole "optional" thing they designed the system
[13:32] drbobbeaty for.
[13:40] nicolas I see
[13:40] nicolas RPC was deprecated for example
[13:41] nicolas I'm relativeley new to the API so no version changes yet since my first install
[13:42] drbobbeaty You should be OK with PB, if you like it. Just be aware that switching versions of PB is often a large-scale upgrade.
[13:43] nicolas I will take a look into MessagePack as well, but for now i'll continue to use C structs
[13:44] nicolas I saw that zmq guys use "gsl" also, but I don't know if it is what I really need
[13:49] drbobbeaty There are a lot of serialization schemes -- each project has it's own needs, etc. No one right tool for all jobs.
[13:52] sustrik nicolas_: take care with PODs
[13:52] sustrik POD binary layouts are not consistent across microarchitectures
[13:53] nicolas that is correct
[13:53] sustrik say between big endian and little endian machines
[13:53] nicolas in this particular project, we are tied to x86_64
[13:53] nicolas intel
[13:56] sustrik you should use the same OS as well
[13:56] sustrik but it's pretty fragile anyway
[13:57] lupine_85 a simple (de)serialization scheme shouldn't take *that* long to knock together and is well worth it, as a general rule
[13:59] nicolas if I implement my own serialization scheme it will surely be slower that any scheme out there
[14:00] lupine_85 well, offhand I'd always recommend grabbing a format that already works
[14:01] lupine_85 (hence mentioning json :))
[14:01] nicolas as I said, I'm using PB for non performance critical tasks, but I had no time of doing benchmarks
[14:01] nicolas or measuring
[14:05] nicolas sustrik: any recommendation ?
[14:06] sustrik what language are you using?
[14:07] nicolas C++
[14:07] mkoppanen hmm, bladecenter is showing no love at the moment
[14:08] mkoppanen there is "Fault LED" turned on the front panel
[14:08] mkoppanen but that's about all diagnostics i've found this far
[14:14] sustrik :(
[14:14] nicolas sustrik: I'm using zmq cpp bindings with C++ and some C
[14:14] sustrik nicolas_: if the datatypes you are using are simple, possibly the easiest solution is to serialise it by habd
[14:14] sustrik hand
[14:15] sustrik e.g. htons, htonl
[14:15] sustrik if it's more complex, i would probably go for PB
[14:16] nicolas isn't htons and htonl overkill for inproc simple datatypes?
[15:09] alanw123 hi, i hope someone here can help me with what i hope is a simple question
[15:09] alanw123 i have a daemon that receives and forwards zmq messages (not interesting)
[15:10] alanw123 and i'd like to integrate a simple monitoring XPUB socket that would act
[15:10] alanw123 as a tap that other processes could subscribe to in order to receive some
[15:10] alanw123 real time statistics
[15:10] alanw123 my question is: is it necessary or required that you should recv subscription requests from an XPUB socket?
[15:11] calvin it's not necessary nor required
[15:11] calvin the subscriptions are managed internally
[15:11] calvin i'm fairly certain you *can* call recv on an XPUB socket, but i use XPUB/XSUB pattern extensively and never have
[15:12] alanw123 calvin: ok, thanks
[15:12] alanw123 so i don't have to worry about buffers getting full, etc
[15:13] calvin correct
[15:13] calvin you can set a RCVHWM on the publisher if you're worried it won't be able to keep up
[15:13] alanw123 right - so how long do these sub scription messages hang around for?
[15:13] calvin hmm, i'm not entirely certain
[15:14] alanw123 ok, i think i'll try a few things
[15:15] mkoppanen i assume you get ENOTSUP on XPUB recv
[15:21] sustrik alanw123: the subscription messages are buffered in XPUB socket
[15:21] sustrik if you are not reading them, the memory usage will gradually grow
[15:22] sustrik if you are not interested in subscriptions, use simple PUB socket
[15:22] sustrik it applies the appropriate filters internally and drops the subscription without delivering it to the user
[15:26] calvin what if you care about publisher side filtering but don't specifically care about what the clients are subscribing to?
[15:27] alanw123 sustrik: ok, that's interesting - i was assuming that a zmq3 PUB socket operated the same way as zmq2
[15:27] alanw123 in that a zmq2 PUB socket will not perform any filtering
[15:27] calvin wait, PUB/SUB in zmq3 does publisher side filtering, not client side?
[15:31] sustrik right
[15:31] alanw123 sustrik: great!
[15:32] sustrik to understand the design, have a look at this:
[15:32] sustrik http://www.250bpm.com/concepts#toc5
[15:33] sustrik just substiture REQ/REP/XREQ/XREP for PUB/SUB/XPUB/XSUB
[15:39] calvin Weird, the 3-0 API says that SUB socket sets sock opt ZMQ_SUBSCRIBE and "Newly created ZMQ_SUB sockets shall filter out all incoming messages, therefore you should call this option to establish an initial message filter."
[15:40] calvin Does this mean that you're supposed to send a message with a SUB socket to a PUB socket so that the PUB socket knows about the subscription?
[15:40] calvin and not set sock opt ZMQ_SUBSCRIBE?
[16:13] cremes calvin: no, it doesn't mean that at all; you cannot send messages with SUB sockets
[16:13] cremes you must call zmq_setsockopt with ZMQ_SUBSCRIBE to set a filter because by default a SUB socket ignores all messages
[16:14] cremes with the new 3.0 api setting a filter has the effect of the underlying library generating a subscription message and sending
[16:14] cremes it to the PUB socket, but that is *not* visible to your application code
[16:14] calvin interesting
[16:14] cremes this is how 3.0 does publisher-side filtering
[16:14] calvin i wish the documentation on this was less confusing; the API docs make it seem like the SUB socket is still filtering in 3-0
[16:15] cremes the SUB sockets have to transmit their filter lists to the PUB socket
[16:15] calvin right
[16:15] cremes calvin: for the pgm transport the SUB socket *is* still filtering
[16:15] calvin i see that now in zmq::sub_t::xsetsockopt
[16:15] calvin oh
[16:15] cremes this publisher-side filtering only works for point-to-point protocols
[16:16] cremes yep, tricky stuff... but i bet you have it now :)
[16:18] calvin true i'm just wondering if make the API should be updated to reflect it so it's clearer for future users
[16:18] calvin if maybe*
[16:19] cremes if you are confused, then it could probably use some clarification
[16:19] cremes feel free to fork it, write up your improved man page and submit a pull request
[16:19] cremes (there's a process to follow... check the zeromq.org site for the steps)
[16:20] calvin ok thanks
[16:24] louisn Looking for anyone with experience using the java binding in an applet.
[16:27] Steve-o sounds complicated as it is JNI based
[16:29] Steve-o You really want to look at something HTTP or websocket based to say mongrel and have that as a 0mq broker
[16:34] louisn It is complicated further that I'm interested in using it client side only as a message transport on the workstation between peripherals and the browser tier of the application.
[16:35] louisn So HTTP or websocket doesn't apply.
[16:35] louisn The java binding looks to have given some forethought to the applet by how it attempts to load the native library.
[16:47] louisn I'm wondering how to use the EmbeddedLibraryTools with the applet.
[16:50] Steve-o You might have to ask on the mailing list or try the jzmq GitHub tracker: https://github.com/zeromq/jzmq/issues
[17:42] mkoppanen motherboard changed on the failed build cluster blade
[17:42] mkoppanen it should be operational tonight (hopefully)
[17:44] mkoppanen the RAID has partially failed apparently but it remains to be seen whether the data is still there
[17:59] calvin there is one interesting thing i just noticed though in sub.cpp
[17:59] calvin it says
[18:00] calvin "// Switch filtering messages on (as opposed to XSUB which where the filtering is off"
[18:00] calvin and options.filter is true
[18:00] calvin but if it's filtering on the PUB side, in both PUB/SUB and XPUB/XSUB, shouldn't options.filter be false?
[18:23] mkoppanen i think filtering is still on sub side if you use PUB/SUB
[18:23] mkoppanen XPUB/XSUB for server-side filtering
[19:01] deam what could be wrong if a message send from a XREP connection is not received by a XREQ connection?
[19:01] deam the other way around works fine
[19:02] calvin mkoppanen: that's what i thought, but sustrik says that is not true in 3.0
[19:02] calvin and i looked at the code and it seems that in sub.cpp it does send a message when you call setsockopt with ZMQ_SUBSCRIBE
[19:15] deam there is no routing done in between, should I send a routing packet anyway?
[19:18] deam never mind
[19:20] deam hey sundbp_, didn't get to thank you.. but thanks for the majordomo page, really got me in the right direction!
[19:20] sundbp deam: cool
[19:21] sundbp deam: i liked it myself. put together a ruby implementation
[19:21] deam I am observing the Python implementation
[19:21] deam trying to reproduce it using txZMQ
[19:22] sundbp deam: oh ok. when i was reading no python there. i read the haxe one. a language i never heard of but it's very readable.
[19:23] deam haxe? never heard of either :)
[19:24] sundbp deam: got a good 0mq binding though :) and all examples of the guide pretty much
[19:26] deam ah, well 3 years ago I chose Python as my primary language
[19:26] deam not going to change that anymore :)
[19:37] hcles www.cat/quit
[19:37] hcles oops
[19:37] sundbp deam: just meant if you don't find soem examples in the guide in python the haxe ones are almost all there and easy to read (denser than the C ones)
[21:01] przemeklach hi, I've installed 0MQ as per the UNIX instructions and I downloaded the zguid; I'm trying to compile hwserver.c but I get the following errors
[21:02] przemeklach hwserver.c:23: error: too few arguments to function ‘zmq_recv’ hwserver.c:34: error: too few arguments to function ‘zmq_send’
[21:02] przemeklach not sure if my install is wrong
[21:02] przemeklach some help would be appreciated, thanks in advance.
[21:04] cremes przemeklach: the function signature for zmq_send() changed from version 2.1.x to 3.x
[21:04] przemeklach I get this error when i run gcc hwserver.c or build hwserver.c
[21:04] cremes which version of the library did you install?
[21:04] przemeklach 3
[21:04] przemeklach ok
[21:04] cremes ok
[21:04] przemeklach so the guides are not up to date yet
[21:04] cremes that's the issue
[21:04] przemeklach ok
[21:04] cremes correct
[21:05] przemeklach ok tks for your help
[21:10] przemeklach hm, ok; so I'm new to zeroMQ; do you recommend that I maybe use v2.1 so that it's easier to learn?
[21:10] przemeklach I'm guessing the documentation for 3 won't be that great until it's out of beta?
[21:11] cremes przemeklach: if you are brand new, i suggest learning the concepts using 2.1.x
[21:11] cremes all of those concepts will transfer over to 3.x
[21:11] przemeklach ok tks
[21:11] przemeklach ok
[21:11] przemeklach is there a release date for 3?
[21:12] cremes it's already released
[21:12] przemeklach ok
[21:13] przemeklach any idea on when documentation will be updated?
[21:13] przemeklach I ask because I'm working on a 2 yr project
[21:13] cremes the man pages are already updated
[21:13] cremes all of the examples and the guide may take a while to get updated
[21:13] przemeklach and I'm evaluating zeromq right now for our publish subscribe architecture
[21:14] cremes if you're on a 2 year project, you should use 3.x
[21:14] przemeklach ok
[21:14] cremes but again, learning how to use the lib and understanding its concepts can be done
[21:14] cremes with 2.1 and all of the example code that already works
[21:14] cremes perhaps as a learning exercise you would like to fork the documentation and update it for 3.x?
[21:15] przemeklach ok i will start with 2.1; i guess it should be good enough for evaluation purposes
[21:15] przemeklach update documentation
[21:15] przemeklach hm...
[21:15] przemeklach well I would love to contribute and I will if I can
[21:16] przemeklach i'm guessing no one is really working on that right now?
[21:18] cremes przemeklach: it's hard to say... someone may have forked it and started already
[21:18] cremes i just know that i am not working on it :)
[21:32] grantr cremes, that brings up a question for me too: if i'm going about adding distributed actors to celluloid via zeromq, should i be using zeromq 3?
[21:32] grantr in particular i'm considering using dealer/router and it appears that may have changed in 3.x
[21:32] cremes grantr: yes, and you should also point me to your project so i can follow it :)
[21:33] grantr cremes, very early days yet but i have some ideas i have been ruminating on :)
[21:33] cremes i was considering doing the same thing to celluloid
[21:33] grantr sometimes i wonder if i'm just implementing erlang in ruby
[21:33] cremes i think tarcieri would say yes
[21:34] cremes but with sane syntax
[21:34] grantr is the ffi-rzmq 3.x support working?
[21:34] grantr hopefully
[21:34] cremes it is working on the master branch
[21:34] cremes i haven't pushed a new gem yet with 3.x support in it
[21:34] grantr ok
[21:35] cremes i'm still updating example code in other projects so they follow the new api (i changed the api
[21:35] cremes somewhat significantly in the master branch)
[21:35] grantr ah good to know
[21:36] cremes yes, it follows the C api much more closely now
[21:37] cremes it isn't very ruby-ish as a result, but wrapping it with nice ruby idioms should be much easier
[22:07] mkoppanen cremes: are you building on rzmq?
[22:07] mkoppanen or completely different?
[22:08] cremes mkoppanen: it's been a completely different project since the beginning; it uses ffi instead of the C extension api
[22:08] mkoppanen cremes: i could add that to daily builds as soon as hardware is back up
[22:09] cremes sounds good; what do i have to do to get that to work?
[22:09] mkoppanen just provide github url
[22:09] mkoppanen i usually add 2.1, 3.0 and master builds
[22:10] mkoppanen currently things are a bit wonky as the blade that hosts the build cluster died
[22:10] cremes https://github.com/chuckremes/ffi-rzmq
[22:11] cremes mkoppanen: what does the build server do? can it run my specs using the latest c lib build?
[22:11] mkoppanen cremes: yes, it can do just about anything
[22:12] cremes great
[22:12] cremes sign me up
[22:12] mkoppanen currently it builds: libzmq, zerom2-1, zeromq3-0, php ext, rzmq, erlzmq, jzmq, clrzmq, libczmq and some others
[22:12] mkoppanen and tests where available
[22:12] mkoppanen and creates snapshot RPMs and source snapshots
[22:13] mkoppanen hopefully daily windows DLLs soon as well
[22:13] mkoppanen well, currently it doesn't build anything as the hardware failed completely today
[22:13] cremes good, i use libzmq on osx, linux & windows every day
[22:13] cremes heh
[22:13] mkoppanen but motherboard has been replaced
[22:14] mkoppanen and it's waiting for vmware esxi install
[22:14] mkoppanen unfortunately it's a bit late where the box is located so most likely tomorrow
[22:14] cremes fine by me... i'll be around
[22:15] mkoppanen OSX dmgs would be nice as well
[22:15] mkoppanen but that's a bit harder to organise
[22:16] cremes why would you need a dmg?
[22:17] mkoppanen mac os x binaries
[22:18] cremes i assumed your build box was linux
[22:19] mkoppanen i got linux, freebsd and solaris
[22:19] mkoppanen hopefully windows soon if i can find a license from somewhere
[22:19] mkoppanen mac would be nice but probably requires mac mini or something colocated
[22:20] cremes right; there are probably enough users where that would be worthwhile
[22:20] mkoppanen yeah, just shame to buy brand new mac mini to put into datacentre :)
[22:21] cremes heh
[22:22] geekstuff hi. trying to install zeromq via pear .... pear.zero.mq server cannot be reached ... anyone know if it is now deprecated ?
[22:23] cremes geekstuff: it's down right now; ask mkoppanen for details
[22:23] geekstuff thanks @cremes
[22:48] przemeklach i've installed zeromq v2.1 but when I try to build hwserver I get the following error: /usr/bin/ld: cannot find -lczmq
[22:49] przemeklach ld does exist
[22:54] taotetek hi gang
[22:54] mkoppanen przemeklach: have you installed libczmq?
[22:55] mkoppanen hi taotetek
[22:55] przemeklach i have to install that ontop of zeromq
[22:55] przemeklach ?
[22:55] mkoppanen przemeklach: yes
[22:56] przemeklach and libczmq is the client
[22:56] przemeklach right
[22:56] mkoppanen libczmq is higher level C binding for zeromq
[22:56] mkoppanen it adds some abstractions to reduce boiler-plate
[22:57] przemeklach I thought that the C & C++ bindings came automatically with zeromq install
[22:57] przemeklach or is that just with version 3?
[22:58] przemeklach according to here http://www.zeromq.org/intro:get-the-software
[22:58] przemeklach at the bottom it says
[22:58] przemeklach You get C and C++ bindings automatically with ØMQ.
[22:58] mkoppanen przemeklach: as i said, czmq is higher level binding
[22:58] przemeklach ok
[22:58] przemeklach got it
[22:58] przemeklach tks
[22:58] mkoppanen the low-level ones ship with zeromq package
[22:59] mkoppanen but usually you end up writing a lot of boiler-plate code with those
[22:59] mkoppanen Turning this wishlist into reality gives us czmq, a high-level C API for ØMQ. This high-level binding in fact developed out of earlier versions of the Guide. It combines nicer semantics for working with ØMQ with some portability layers, and (importantly for C but less for other languages) containers like hashes and lists.
[22:59] przemeklach awesome
[22:59] przemeklach tks
[23:04] pieterh taotetek: mkoppanen: hi guys
[23:28] mkoppanen yyyeeehaaa
[23:40] CIA-79 pyzmq: 03MinRK 07 * rd4d28b7 10/ .mailmap : add mailmap - http://git.io/cqTnhg
[23:50] nanodocumet Hi, I have a quick question, which pattern is best to do batch sending of emails? Web framework will add email to send to a queue and then a worker will process the request (only 1 worker, only 1 time) to send the email.
[23:51] mkoppanen nanodocumet: you probably want the web script to receive ACK that the message has been received for processing?
[23:52] mkoppanen client |DEALER| <-> |ROUTER| BROKER |DEALER| <-> |ROUTER| WORKER
[23:52] mkoppanen might work for you
[23:52] nanodocumet not sure. Do you mean that the web script knows that the email is being sent? something like what is the current status of the request?
[23:52] mkoppanen no, i mean let's say the queue fails to handle the message
[23:52] mkoppanen do you need to know that synchronously?
[23:53] mkoppanen or more of a fire and forget
[23:53] nanodocumet no, async.
[23:53] mkoppanen you could use PUSH socket on client and PULL on the other end
[23:53] mkoppanen that's the simplest scenario
[23:54] nanodocumet yes. Looking for something simple to, first learn zeroMQ.
[23:54] nanodocumet web framework send a message to a queue.
[23:55] mkoppanen have you taken a look at the zeromq guide?
[23:55] nanodocumet A worker grabs the message and process it (this case does email sending, but could be other tasks as well).
[23:56] mkoppanen the guide has a few patterns for handling worker processes
[23:56] nanodocumet yes, I have, but wasn't sure if request-reply or publish-subscribe or any other was better.
[23:56] mkoppanen well, it all depends on the use-case
[23:57] mkoppanen publish subscribe is usually one publisher, multiple subscribers
[23:57] mkoppanen probably not the most suitable pattern for worker processes
[23:57] mkoppanen as each one of the workers would receive the same message
[23:57] mkoppanen you could didive the messages into topics for each worker but it gets messy if you add / remove workers or they die
[23:57] nanodocumet Example: user wants to register to website. After registration user is shown with a message of wait to receive email with extra info.
[23:58] mkoppanen for that scenario i would use DEALER on the client and ROUTER on the broker as mentioned before
[23:59] mkoppanen or PUSH/PULL, depends on the scenario and the rest of the infra
[23:59] mkoppanen but for registration email i would like to receive ACK from the queue saying "it will be processed" or "dude, cannot do this atm"
[23:59] mkoppanen like let's say all worker processes are down. i might want to tell the user on the website that the email might take 10 - 15 minutes

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy