Thursday March 17, 2011

[Time] NameMessage
[00:01] believa What tag/branch to use of the zeromq github repository to get the same source as the v2.0 stable release on the zeromq website?
[00:03] believa actually - should be the v2.0.10 tag I'm guessing.
[00:14] tokyodan Trying to install in Ubuntu 10.04 LTS server. I followed all the instructions. did './configure'. Then when I execute 'make' I get 'No targets specified and no makefile found'. What is wrong?
[00:14] tokyodan all the LINUX dependencies have also been installed.
[00:17] bpl29 My make check is failing on OS X. It's failing on test_shutdown_stress
[00:19] bpl29 I'm reading there is an assertion error in mailbox.cpp, and in fact it's commented in the source to say that this assertion should never fail.
[00:20] tokyodan bp129 doesn't seem to be much help forthcoming here.
[00:21] bpl29 right, I'm wondering if people have actually installed on OS X.
[00:21] bpl29 There is a developer thread about this particular error, but it's dated
[00:22] bpl29
[00:22] bpl29 Perhaps someone has gotten about this.
[00:22] Guthur bpl29, I know someone was running on OSX today
[00:23] Guthur and indeed I have a user of clrzmq2 who runs on OSX
[00:23] tokyodan bp129 I'm trying on Ubuntu 10.04 server now with not much luck. I'm afraid to install on my Snow Leopard machine right now.
[00:23] Guthur I don't use it myself so I can't really help
[00:23] Guthur tokyodan, zeromq built fine on ubuntu 10.04 for me recently
[00:23] tokyodan thanks bp129
[00:23] Guthur which version are you using
[00:24] Guthur I wasn't running server mind you
[00:25] bpl29 I just downloaded from git
[00:25] tokyodan What causes make to error out with ''No targets specified and no makefile found' ?
[00:25] tokyodan Guthur Did you just type 'make' with no arguments?
[00:25] Guthur you ran autogen first, right>
[00:26] Guthur ?
[00:26] tokyodan autogen> no. Didn't see that explained anywhere.
[00:26] Guthur I think I ran autogen first then configure then make
[00:27] tokyodan what does autogen do? And does it take arguments?
[00:27] Guthur I think it should try to automatically detect settings for you platform
[00:27] tokyodan I'm using the latest version of OMQ 2.1 (rc2)
[00:28] tokyodan I thought that's what ./configure also did.
[00:28] tokyodan did you run autogen on OS X or Ubuntu?
[00:28] Guthur Ubuntu
[00:29] Guthur you are right though
[00:29] Guthur configure should do it
[00:29] tokyodan Yeah. frome its output that's what it looked like it was doing.
[00:30] Guthur my make fu is pretty weak tbh
[00:30] tokyodan Seems like it should have created a makefile of something too, but didnt. although I got and
[00:31] Guthur i'm just downloading fresh here to check
[00:33] Guthur you must have ran autogen at some point
[00:33] Guthur I have no ./configure in a fresh download
[00:34] Guthur tokyodan, the configure completes ok?
[00:35] cremes if you get a mailbox assertion on OSX, you need to do some kernel tuning
[00:35] cremes search the site for the 'tuning guide'
[00:35] Seta00 anyone here using clrzmq2?
[00:35] cremes it has an entry on how to expand the socket buffers used by the kernel
[00:35] Guthur Seta00, I do
[00:36] Guthur Seta00, any issues?
[00:36] cremes bpl29: look at the tuning guide
[00:36] Seta00 Guthur, it seems you can't detect EAGAIN when using NOBLOCK on a recv call.
[00:36] Seta00 so I guess I'll have to use polling?
[00:37] bpl29 cremes: where is the tuning guide located?
[00:38] Guthur Seta00, if there is no msg the method just silently returns
[00:38] cremes not sure... try the search box
[00:38] Seta00 bpl29,
[00:38] bpl29 thanks Seta00
[00:38] cremes bpl29: that's it! ^^
[00:38] bpl29 thank you both
[00:38] cremes on osx the default buffers are all too tiny :(
[00:38] Seta00 bpl29, for future reference: Guthur, yeah but silently returning doesn't
[00:38] Seta00 oops
[00:38] Seta00 wrong ctrl-v
[00:38] Seta00
[00:39] Seta00 Guthur, so I should check the output buffer?
[00:39] cremes bpl29: btw, i run on osx all the time; once you get some tuning out of the way 0mq works really nicely on it
[00:39] Guthur Seta00, the return byte array will be null
[00:40] bpl29 Okay, great. I'll give it a shot.
[00:40] Guthur Seta00, I didn't feel it was right to throw an exception for it
[00:41] Seta00 what how did I miss that
[00:41] Guthur it should be the only time you receive a null msg
[00:41] Guthur and empty message being byte[0]
[00:41] Seta00 yeah, I see it now
[00:41] Guthur Seta00, cool, does that behaviour seem sane to you
[00:41] mkent Noob question: If you have zeromq set to queue to disk, is there any API for monitoring the disk queue?
[00:42] Guthur I'm always looking for feedback
[00:42] Seta00 Guthur, yes it does.
[00:42] Seta00 my current zmq code is ridiculous, so I'm doing some refactoring :)
[00:43] tokyodan Guthur the ./configure completed with 'error: unable to find a working c++ compiler'.
[00:43] Guthur tokyodan, yeah that would be a show stopper, hehe
[00:44] Seta00 tokyodan, sudo apt-get install build-essential g++
[00:44] tokyodan I thought Ubuntu would have a C++ compiler built in.
[00:44] Seta00 :P
[00:44] Seta00 tokyodan, only comes with a C compiler
[00:44] tokyodan thanks Seta00
[00:44] Seta00 yw
[00:45] Guthur Seta00, if you ever have any issues with clrzmq2 you can usually catch me around here, or raise an issue on github, which I'll try to get asap
[00:45] Seta00 Guthur, awesome, I'll keep that in mind
[00:46] Guthur I'd really like to clean up some of the API, but I'm waiting until ZMQ 3.0
[00:48] Guthur Seta00, Are you targetting Mono perchance?
[00:49] Seta00 Guthur, yes
[00:50] Guthur make sure to grab the most recent version of clrzmq2
[00:50] Guthur I found a signalling issue in relation to the Mono GC
[00:50] tokyodan Guthur & Seta00 Thanks you guys. You got me through 'make'. It's running right now.
[00:50] Guthur you may have seen it on the mailing list recently
[00:50] Seta00 nice :)
[00:50] Guthur tokyodan, cool, happy coding
[00:51] Guthur Seta00, basically signals generated by the Mono GC can cause the syscalls to return EINTR which would throw an exception in clrzmq2
[00:52] Guthur the workaround I decided on was to reenter the call for all signals bar sigint and sigterm, which would instead exit the program
[00:53] Guthur this mainly effected long running syscalls like those used in poll
[00:53] Guthur obviously it was only an issue on POSIX platforms
[00:53] tokyodan 'make' finished with no errors. yeah!
[00:56] tokyodan Well it seems to have installed OK. Now for trying it out. Still have to get the Lua part though.
[02:06] tokyodan I installed the Lua binding on Ubuntu 10.04 server. But when I do: require 'zmq' I get module zmq not found. Seems like make is not putting the files into the proper Lua directories. How do I cure this outside of creating all kinds of symlinks.
[03:39] tokyodan This doesn't work with Lua
[03:39] tokyodan I followed this video:
[03:41] tokyodan And unlink in the video the send("Hello") command returns before I even start up the receiving end.
[03:41] tokyodan I start up the recv loop on the receiving side and it just sits there blocking and waiting for input
[03:47] michelp last night i was musing on the "supercomputer" descriptions in the guide
[03:48] michelp having recently read "The Connection Machine" by D. Hillis it made me think that 0mq would make an excellent interconnect mechanism for various SIMD topologies
[03:50] michelp the guide basically spells out how to do those topologies with a control channel to issue "instructions". in conjunction with something like NumPy it would be an excellent platform for distributed computing
[03:50] michelp anyway, just musing over dinner on the subject ;)
[03:53] tokyodan Does the latest version on OMQ work with Lua 5.1.4?
[03:54] tokyodan on Ubuntu 10.04 LTS Server?
[03:56] michelp tokyodan, I don't use Lua so can't help you specifically, but have you looked at "The Missing Message Problem Solver" at ?
[03:56] michelp it might help you diagnose the problem
[04:04] tokyodan michelp thanks!
[04:05] michelp sure np
[04:06] tokyodan I see that UP/DOWNSTREAM and replaced by PULL/PUSH. Is that correct? I've see examples with PUB/SUB?
[04:10] cremes tokyodan: yes, the UP/DOWNSTREAM names were replaced by PUSH/PULL
[04:10] cremes PUB/SUB has not changed nor are there any plans to change it
[04:25] tokyodan is there a difference between PUSH
[04:25] tokyodan PULL and PUB/SUB?
[04:26] cremes tokyodan: yes, they are quite different
[04:27] cremes in PUB/SUB, any messages sent via PUB are broadcast to *all* subscribers simultaneously
[04:27] cremes in PUSH/PULL, messages send via PUSH are delivered round-robin to all connected PULL sockets
[04:27] cremes if you haven't read the guide yet, please do so
[04:28] cremes there are lots of examples and quite a bit of text explaining this
[04:28] cremes see the channel topic for a link to the guide
[04:28] tokyodan That's what I'm going to do. Thanks.
[04:28] tokyodan I already got it printed out. I was just jumping in, installing, and trying the examples.
[04:33] cremes tokyodan: that's the best way to figure it out
[07:38] sustrik mikko: hi, are you there?
[09:53] pieterh sustrik: ping
[09:53] sustrik pong
[09:53] pieterh I'm sending the OpenPGM patches to the list
[09:53] sustrik great
[09:54] sustrik how was the meetup?
[09:54] pieterh this covers the changes Mikko made but also include the tarballs, so they're large
[09:54] pieterh the meetup was excellent, but too many people
[09:54] pieterh :-)
[09:54] sustrik i can update the tarball myself
[09:54] sustrik no need to include it into the patch
[09:54] pieterh I can't make a patch without the tarball, it's in one commit
[09:54] sustrik ok
[09:54] sustrik np
[09:54] sustrik too many?
[09:54] pieterh yeah, twenty or twenty five
[09:55] pieterh impossible to speak to everyone
[09:55] sustrik oh my, not enough chairs?
[09:55] pieterh we found space, Mikko and Ian organized it very well
[09:56] pieterh i'm testing that Steve's patches apply, will let you know
[09:57] sustrik i can do that
[09:57] sustrik no problem
[09:58] pieterh ok, they apply cleanly
[09:58] sustrik thx
[09:59] pieterh these changes are more complex than normal because they include the tarballs
[09:59] pieterh Mikko wants to make the build system pull the tarballs down dynamically afiau
[09:59] pieterh that would be neat
[09:59] sustrik in the past the tarballs were updated out of normal process
[10:00] sustrik ie. not via mailing list
[10:00] pieterh yeah
[10:00] pieterh this was because I used a pull request from Mikko's git
[10:00] sustrik as for automatic update, how would it know when to update?
[10:00] sustrik each new version?
[10:01] pieterh hmm, at install time?
[10:01] sustrik ah, that way
[10:01] pieterh since 0MQ now builds with any version of OpenPGM
[10:02] pieterh all you need to do is say, "please build 0MQ with OpenPGM vx.y.z" and it would pull and unpack that tarball
[10:02] pieterh something like that, presumably with defaults
[10:02] sustrik i see
[10:02] pieterh this is possible because OpenPGM now has full autotooling, so 0MQ can be much more naive about it
[10:02] sustrik is steve ready to freeze the API?
[10:03] pieterh Steve...?
[10:03] sustrik mc coy
[10:03] pieterh I meant, "Steve... ping...?"
[10:03] sustrik if not, the auto-upload could break
[10:03] pieterh yes
[10:03] pieterh this is speculation, anyhow, but Mikko and Steve already did amazing work on the integration
[10:04] pieterh so I assume they'd get this working properly
[10:05] sustrik ok, let me apply the patch
[10:23] sustrik pieterh_1: you should consult the build system maintainers before committing build-related patches imo
[10:24] sustrik mikko presumably, in this case
[10:24] pieterh_1 you mean Steve's patches...
[10:24] pieterh_1 right
[10:24] sustrik yes
[10:24] pieterh_1 I'll ask Mikko to review them, good idea
[12:04] Guthur pieterh, with peering3 what sort of output should I expect?
[12:20] Guthur oh nvm, working now
[12:30] s0undt3ch hello ppl
[12:30] s0undt3ch can a publisher know where a connection is comming from?
[12:30] s0undt3ch ie, so I can auth/deny a subscription?
[12:35] sustrik no
[13:45] stimpie Iam getting unstable performance using jzmq on windows, 1,000,000 x 128B messages gets 300K/s. 10,000,000 x 128B messages gets 20K/s
[13:46] stimpie it looks like this is caused by the sender, using a c sender and a java receiver gets a stable 300K/s
[13:47] sustrik it must be java binding issue then
[13:47] sustrik try reporting it on jzmq's bug tracker
[13:47] sustrik or on the mailing list
[13:49] stimpie ok I will try to create a simple reproducible scenario. (or just provide the results to my boss, saying we shouldn't use windows ;-)
[13:49] fred86 When using pub/sub sockets that stay open for a long time (without any activity), I find that after a while (maybe 1+ hours later), the client-side stops receiving messages (however when I open a new client, the new client can receive messages). Do the sockets reconnect automatically if there's a short network outage? What could the problem be? Thanks.
[13:50] cremes stimpie: does this example use PUB/SUB?
[13:50] cremes stimpie: if so, your publisher could be overrunning the subscriber; in that case the publisher will queue the messages in memory
[13:50] cremes which could lead to a slowdown if your box doesn't have enough RAM
[13:51] stimpie cremes, yes its a PUB/SUB and memory usages skyrockets
[13:51] cremes stimpie: then it is queueing in memory
[13:51] cremes does the machine running the publisher have enough RAM or is it swapping to disk?
[13:52] stimpie its starts swapping at some point
[13:52] cremes ok, then you have your answer as to why it gets slower
[13:52] stimpie but why doesnt this occur with a c version?
[13:52] cremes which part was in C?
[13:53] stimpie I had a sender and receiver in c and in java
[13:53] cremes c -> c, no problem?
[13:53] cremes c -> java, slow?
[13:53] cremes how about java -> c, does that work?
[13:53] stimpie c->java is ok
[13:53] stimpie java -> c, is slow
[13:54] cremes so which combo produces this slowdown?
[13:54] stimpie java -> java is slow and java -> c is slow
[13:54] stimpie c-> c, c -> java are fine
[13:54] cremes interesting...
[13:55] cremes java -> c being slow doesn't make any sense; that could certainly be a bug
[13:55] cremes does the memory usage of the publisher sky-rocket in that combination?
[13:56] stimpie yes
[13:56] cremes weird... yes, open a ticket with all of these details including the code
[13:57] stimpie I was thinking that maybe java uses some object overhead on each messages causing it to use more memory and start swaping sooner then the c versions
[13:59] sustrik fred86: yes, there's automatic reconnection
[14:00] fred86 hmm, okay. Then this problem is a bit bizarre. the layout is a chatroom-like server (where the server does some processing before publishing a message). if there isn't any activity for a while, I find that the clients stop getting messages from the server.. (but they can still send messages..and I can see the server processing everything fine). and if I open new clients, they will be able to receive messages. do you have any idea wh
[14:01] sustrik no,sorry, try to make a minimal reproducible test case and i'll have a look
[14:02] sustrik stimpie: jave uses gc
[14:02] sustrik java
[14:02] sustrik which could mean that object deallocation is delayed
[14:03] sustrik in that case each message would be stored in memory twice
[14:03] sustrik once in 0mq buffer
[14:03] sustrik second time in form of gargage on java heap
[14:05] stimpie I will do some more test to see if it really is only a memory issue.
[14:43] gambi how do I return a message via XREQ in a stream callback? on one side i've got this zmqstream and a on_recv() callback and on the other side a regular socket which does the send() (that works) and the recv() (which doesn't)
[14:43] gambi XREQ/XREP
[14:44] gambi unfortunately I'm not sure how else do explain this, sorry :(
[14:48] gambi basically I'm trying to send build something closely related to this: but I'd like to have some processing done before it returns the message to the server.
[14:49] gambi sorry for the random words that mix into my sentences ... I guess I'm already quite tired ;)
[14:50] guido_g simply send the message back
[14:52] guido_g gambi: see
[14:55] gambi guido_g: oh, great thanks for that! i'll try that right away. :)
[14:56] guido_g one should try the products documentation *before* using the product...
[14:56] gambi guido_g: oh, c'mon ... I've been reading the docs for hours ... ;)
[14:56] guido_g sure
[15:21] xs suppose the rep side of a reqrep dies after the req side has sent it's request. then the rep side recovers. shouldn't the req side re-send its request? this doesn't seem to be happening for me (i'm using multipart messages if that changes things)
[15:26] cremes xs: no, 0mq doesn't auto-resend anything
[15:26] cremes you would need to timeout the send and retry yourself
[15:27] cremes to do so you have two choices and both use zmq_poll()
[15:27] cremes 1. close the REQ socket, reopen it and resend
[15:27] cremes 2. use an XREQ socket (this doesn't enforce the strict req/rep/req/rep message pattern) and resend upon timeout
[15:27] cremes does that make sense to you?
[15:28] xs i think so. thanks.
[15:28] xs am i correct in thinking i just have to timeout?
[15:28] xs there's no way for me to determine that an endpoint has disappeared and reconnected
[15:28] Guthur ch4 for of the guide covers reliability
[15:28] Guthur It might have some useful patterns
[15:29] cremes xs: correct, there is no way to determine that an endpoint has bounced
[15:29] xs ok, thanks, i'll take a look.. i had just assumed 0mq would do this for me. sorry for the noise
[15:29] cremes chapter4 of the guide is a good suggestion
[16:15] stimpie A follow up on my java -> c performance issues, when limiting the send rate I could push my message rate to a ~300K/s, going above that results in swapping. So cause is memory and not a bug in the java wrapper
[16:16] stimpie Still need to figure out how to handle such an overload in production but thats a different subject
[16:20] sustrik use hwm/swap
[16:29] Guthur pieterh, when simulating a server crash with lazy pirate client/server, should the client be generating...
[16:29] Guthur Bad file descriptor
[16:29] Guthur nbytes != -1 (mailbox.cpp:242)
[16:34] Guthur oh sorry nvm, my mistake
[16:59] michelp my coworkers have returned from pycon, it's hard to know if what they are saying about 0mq is FUD they acquired from someone else or not
[16:59] michelp for example, they are saying in the PUB/SUB case that the subscriber is guaranteed to drop the first message always, all the time
[17:00] michelp is that true? i remember seeing something about it in ch 1 but lost track of the problem in my reading of the guide
[17:09] sustrik no, it's not true
[17:10] michelp it seems to be a meme that has dug in hard from pycon. sigh
[17:10] ianbarber it's because if you right a bit of test code, and start publishing and subscribing at the same time, you'll likely push a bunch of messages before the subscriber is finished connecting
[17:11] ianbarber this is a common new-to-0mq issue
[17:13] michelp I suspect this is the part of the guide that has formed the meme: "Even if you start a subscriber, wait a while, and then start the publisher, the subscriber will always miss the first messages that the publisher sends."
[17:14] michelp they read that far, and stop, and don't go on to read the solution
[17:14] michelp Perhaps the very next sentence should be something along the lines of "Keep reading and we'll explain how to do it right" or something like that
[17:17] michelp i think the fact that the sentence is bolded also make it stick in someone's mind more so than all the other information there. anyway, it's unfortunate that it seems like these wrong impressions have spread through such a big community so quickly.
[17:17] michelp maybe we need an anti-FUD faq
[17:18] kdj Yeah, thats too bad. I haven't seen anything like that in my testing.
[18:45] Guthur pieterh, ping?
[19:08] Guthur private_meta, you about
[20:21] mikko hi
[20:25] Guthur hey mikko
[20:25] Guthur I see the meet up at London was a success
[20:25] Guthur you were there, correct?
[20:26] mikko yes
[20:26] mikko i will never drink beer again
[20:28] Guthur stick to coffee, it actually can improve productivity, instead of the opposite, hehe
[20:28] Guthur some nice topics though
[20:29] Guthur the JMS is an interesting one
[20:29] Guthur Just had this situation at work recently, not really related to 0MQ, more that someone wanted to interact with JMS outside of Java
[20:30] Guthur I suggested a thin Java wrapper and 0MQ as the glue
[20:35] Guthur mikko, did you implement any of the examples in ch4 perchance?
[20:42] mikko Guthur: nope
[20:45] mikko JMS came from Java guys at the meetup
[20:45] mikko they said that Java market is a lot easier to penetrate with JMS compatible messaging
[20:48] Guthur yeah they're probably right
[20:49] Guthur I think sustrik suggestion is the sanest one
[20:49] Guthur but probably not enough for the Java community, they like everything to be fully in the Java ecosystem
[20:53] Guthur it'd be an interesting project actually
[20:57] dahankzter I dont know, just a bridge would go a long way
[20:57] dahankzter that way old IBM MQ's can be used in conjunction with nw =MQ stuff
[20:58] dahankzter s/nw/new
[21:02] Guthur yeah, I'm probably being a little cynical
[21:05] dahankzter no, not really. its very much so but just being able to connect your new cool stuff woth the old expensive monster could go a long way
[21:20] sustrik the nice thing about the bridge is that it's cheap
[21:20] sustrik can be done in couple of days, i would say
[21:23] dahankzter perhaps the problem is vendor specific protocols?
[21:23] dahankzter jms does not define any such for the wire at least
[21:26] dahankzter I think AMQP would be a nice place to start
[21:30] dahankzter i think you know lots of this though :) iMatix is mentioned on the AMQP site
[21:48] sustrik dhankzter: you can bridge basically any protocol or API that provides PUB/SUB and/or REQ/REP semantics
[21:49] sustrik as for AMQP there's 0mq/RabbitMQ bridge
[21:52] dahankzter yes sure. i was just thinking from my perspective where normally there is a big proprietary thing eg MQ as "standard". I would not be allowed to pull in an extra middleware thing like RabbitMQ as well as my new code+zeromq. the convincing the management factor is not negligible unfortunately.
[22:14] advant How does zeromq guarantee message persistence and delivery without a database? Or does it use something like hypersonic in-memory database?
[22:25] dahankzter it doesn't
[22:29] cremes advant___: it doesn't guarantee either of those things; what led you to think that?
[22:36] advant oh i dunno, I guess that would be part of a broker anyways