Wednesday May 4, 2011

[Time] NameMessage
[10:38] pieterh Bartzy|work: depends...
[10:38] pieterh what's up?
[10:39] pieterh go for it
[10:40] pieterh ok
[10:41] pieterh have you read the Guide?
[10:41] CIA-75 libzmq: 03Tore Halvorsen 07master * r1012021 10/ src/xrep.cpp : Accessing an iterator after it is accessed is not valid. ...
[10:43] pieterh it doesn't really matter
[10:43] CIA-75 libzmq: 03Martin Sustrik 07master * r3ef7c74 10/ AUTHORS : Tore Halvorsen added to the AUTHORS file ...
[10:43] pieterh whatever is easier for you
[10:43] pieterh obviously, yes
[10:43] pieterh there are a number of patterns you can use for this
[10:44] pieterh check the LRU queue patterns, for a start
[10:44] pieterh you can make the workers each a single process that is single threaded
[10:44] pieterh you can make the workers threads in a single multithreaded process
[10:45] pieterh read the LRU queue pattern & description, plz
[10:45] pieterh search for lruqueue if in doubt
[10:45] mikko Bartzy|work: why do you use gimp?
[10:45] mikko just out of curiosity
[10:45] mikko Bartzy|work: imagemagick tends to be more geared towards headless processing
[10:45] pieterh Bartzy|work: most people would probably use imagemagick except it's harder to spell
[10:46] mikko Bartzy|work: limited how?
[10:46] mato ImageMagick is also sloooow
[10:46] mikko mato: graphicsmagick then
[10:46] mikko Bartzy|work: pretty much
[10:47] mikko and probably more than in gimp
[10:47] mikko last time my trial with gimp ended to not having proper CMYK support
[10:47] mikko Bartzy|work: what do you do in the gimp scripts?
[10:48] mikko what kind of effects?
[10:48] mikko programmatically?
[10:48] pieterh Bartzy|work: I'd suggest initially making one process per gimp server, single threaded, and use an lruqueue device to connect workers to clients
[10:49] mikko cool
[10:50] sustrik hi mikko
[10:50] mikko hi
[10:50] sustrik any chance to check patches at the build system?
[10:50] mikko sustrik: sure
[10:50] mikko sustrik: which ones?
[10:50] pieterh Bartzy|work: nope, all messaging is just 0MQ done poorly
[10:50] sustrik i've tried to fix the issues by giess
[10:50] sustrik guess
[10:51] sustrik but obviously that does not work :)
[10:51] pieterh Bartzy|work: well, I'm assuming you haven't actually tried 0MQ
[10:51] sustrik mikko: well, the sun studio and icc
[10:53] pieterh Bartzy|work: it's not about coolness, if you haven't actually written a couple of 0MQ apps, you can't understand it
[10:53] pieterh so my advice is to go off, make a simple workload distribution architecture, in PHP
[10:53] pieterh it'll take you a day, maybe
[10:53] pieterh if you have questions, of course ask for help
[10:53] pieterh but there are full, complete PHP examples for this
[10:54] pieterh sigh
[10:54] pieterh :)
[10:54] pieterh start by learning 0MQ, then when you know what you want to make, and how, decide how to really make it
[10:55] pieterh it's pointless to debate "what language" when you don't even know what architecture
[10:55] pieterh "ah, but do you think I should use vim or emacs?"
[10:55] pieterh not relevant
[10:55] pieterh first learn some basic messaging patterns, then decide how to make them work for you
[10:56] pieterh it's wise to make a proof of concept *as rapidly as possible* and then throw it away and start again
[10:56] pieterh does this help?
[10:57] djc Bartzy|work: the point is that ZMQ is very good at having something built in a day
[10:57] djc instead of in a week
[10:57] djc so researching it is less valuable than for some other technologies
[10:59] djc the same as what you do now
[10:59] pieterh Bartzy|work: the only research that counts is actually programming
[10:59] pieterh asking others to make your technical choices for you isn't profitable
[10:59] pieterh 0MQ makes it *really easy* to go off and learn by doing, so please do so...
[11:00] mikko Bartzy|work: i've been talking with ian barber about php job distribution system before
[11:00] mikko making use of php-fpm for process management
[11:01] mikko Bartzy|work: would not be very hard
[11:02] mikko but lack of time
[11:02] ianbarber it's a decent model, works fine. The tricky thing with whatever you use is maintaining the worker processes existence, the basic messaging is straightforward, and FPM simplifies that.
[11:03] ianbarber otherwise that kind of work distribution is just basic push messaging
[11:05] djc Bartzy|work: read more of the guide, then just spend a day writing a proof of concept
[11:06] sustrik mikko: i mean, can i log on the build box
[11:06] sustrik and try to fix the sun studio and icc problems there?
[11:08] mikko sustrik: it's behind a nat at the moment without public ip address
[11:08] mikko sustrik: i can add you user to a jump host
[11:08] sustrik hm, alternatively, can i send you patches to test?
[11:08] mikko i think i got a box connected to the internet with sun studio
[11:08] mikko if you want i can take a look in a minute
[11:08] sustrik great, thanks
[11:10] mikko
[11:10] mikko the builds succeed at the moment
[11:11] mikko apart from clang which seems to be a compiler thing on amd64 box
[11:11] sustrik hm, strange
[11:11] sustrik yesterday evening there were errors
[11:11] sustrik anyway, if everything works no, no need to panic
[11:11] sustrik now*
[11:13] mikko Minor patch to keep ICC compiler happy (commit: 5e329ba7cac8a52fbbd2c347064c2d9355009022) (detail)
[11:13] mikko Deallocation functions in zmq.h and msg_t class are consistent. (commit: ceb5e1a0734b0c73bd7f74ec5094ae6ad4f9dfc4) (detail)
[11:13] mikko these are the changes in latest build
[11:15] sustrik yep, those are my guess patches
[11:15] sustrik i thought they were already used in the previous build
[11:15] sustrik it seems they weren't
[11:15] sustrik so everyting is ok now
[11:20] mikko cool
[13:58] pieterh Bartzy|work: yup
[13:58] pieterh so you finished your prototype?
[13:58] pieterh :)
[13:59] pieterh I'd start with the very simplest architecture that I could make to test my concept
[13:59] pieterh in that case it should take you only an hour or two
[14:00] pieterh if you are asking my advice on how to make something, don't argue with the advice I give you :)
[14:00] pieterh make a minimal working model
[14:00] pieterh then improve it to solve specific problems you see with it
[14:00] pieterh otherwise, you will spend time solving problems that may be irrelevant
[14:01] pieterh such as... 1 process vs. N processes
[14:01] pieterh ok, so this is what you want to build...
[14:01] pieterh well...
[14:02] pieterh sigh... if you ask and try to answer too many questions at once, you won't get anywhere quickly
[14:02] pieterh please do the following:
[14:03] pieterh - take the lruqueue example
[14:03] pieterh - make a PHP worker that executes request by request, starting GIMP in a shell each time
[14:03] pieterh - make a test client, in PHP that can drive this worker
[14:03] pieterh - start multiple workers in parallel
[14:03] pieterh - see how it works
[14:03] pieterh ... OK?
[14:03] pieterh when you have that working, let's continue the discussion
[14:04] pieterh it is not relevant, to this discussion, how you talk to GIMOP
[14:04] pieterh GIMP
[14:04] pieterh not at least until GIMP has 0MQ built into it
[14:04] pieterh yes
[14:05] pieterh it's pointless to ask me questions about how to use ZeroMQ when you have no experience actually using it
[14:05] pieterh and when there is a 150-page explanation with full examples...
[14:05] pieterh :) guaranteed you will ask different questions
[14:05] guido_g omg
[14:05] pieterh guido_g: help please
[14:05] pieterh it's like asking about sex when you've never tried it
[14:05] guido_g pieterh: there is nothing one can do
[14:06] pieterh "is it true I need to get really drunk before hand?"
[14:06] pieterh "or would it be better to have a long sauna?"
[14:07] guido_g pieterh: people come here (or to other channels) and demand answers
[14:07] pieterh Bartzy|work: I don't want to be critical, it's good that you think ahead, but don't over-think it
[14:07] guido_g of course w/o even the smalles achievements on their side
[14:07] pieterh guido_g: luckily we're a caring and helpful community
[14:08] pieterh but indeed it could be worth raising the barrier a little...
[14:08] guido_g yes, this does help
[14:09] guido_g it helped back in use-net times and it did the last years in an irc channel
[14:09] pieterh Bartzy|work: the best practice is, and I quote Zed Shaw on this, "programming, motherf***r"
[14:10] pieterh if you come here with questions that actually refer to the context we all share, it's much easier
[14:10] guido_g Bartzy|work: if you can't walk, don't even try to run
[14:11] pieterh good deal
[14:12] pieterh If you still see the problem the same way after using 0MQ, you get your money back
[14:12] guido_g oh pieterh while this case is close, there is another question wrt the brussels
[14:12] pieterh tell me
[14:13] guido_g did you made a decision on weather? going to select the clothes...
[14:13] pieterh I've asked for excellent weather
[14:13] guido_g great
[14:13] pieterh but there's a long queue so my request might not get processed in time
[14:13] pieterh it will be warm and sunny
[14:13] pieterh but perhaps only in August or so...
[14:14] guido_g hehe
[14:14] pieterh they're not using 0MQ, obviously...
[14:14] guido_g luckly it's a very short walk from and to the hotel
[14:14] guido_g ømq might have been made in heaven, but it's not used there
[14:14] pieterh lol :)
[14:15] guido_g is there an idea where the warm-up meeting on monday will be?
[14:16] djc what kind of event is there?
[14:16] guido_g
[14:20] djc nice
[14:20] djc hmm, too bad, I probably won't be able to make it this time
[14:21] mato pieterh: can you make a short url for the brussels conf page?
[14:21] pieterh mato: it's been there for ages:
[14:21] mato ah
[14:21] pieterh I put that in every email
[14:21] pieterh :-(
[14:21] pieterh let me put that on the page as well
[14:21] guido_g it's too short :)
[14:21] pieterh BTW anyone can make a short URL at
[14:22] pieterh djc: thanks :)
[14:22] mato ok, that's better
[14:23] mato pieterh: happy with that topic?
[14:24] pieterh mato: you're the channel admin?
[14:24] mato pieterh: one of them, yes
[14:24] pieterh yes, very happy with this, was going to ask for it before but forgot
[14:24] pieterh thanks!
[14:24] mato np
[15:04] cremes mikko: ping; question on compiling under mingw
[15:05] cremes mikko: i run ./configure; make; make install and let it figure out the defaults
[15:05] cremes mikko: the makefile produces libzmq.dll.a and but i do not see any libzmq.dll
[15:05] cremes am i doing something wrong? do i need to pass another switch to configure?
[15:06] mato cremes: look in src/.libs
[15:06] mato libtool will hide the dll there
[15:07] cremes ah ha!
[15:08] cremes any particular reason 'make install' doesn't do the right thing here?
[15:08] mato yes, what would you have it do on windows? :-)
[15:10] cremes copy libzmq.dll to the same place it copies libzmq.dll.a and
[15:10] mato there is no such place on win32
[15:11] cremes well, under mingw it's pretty standard to install to /c/user/local/lib
[15:11] cremes that's at least a good "known" place to copy it from to other locations
[15:11] cremes as required by your windows apps
[15:11] Seta00 windows for /usr/lib is C:/Windows/System32 :P
[15:12] cremes instead of hiding it in src/.libs (i swear i never would have found it there)
[15:12] mato I guess it could create a 'bin' directory on mingw and stick things there
[15:12] mato but AFAIK there is no real standard install path
[15:12] cremes (i should have run 'find /c -name libzmq.dll"
[15:12] mato other than %WINDIR%\system32 and you don't want that :-)
[15:14] cremes got my first 0mq program running on windows! thanks for your help!
[15:36] djc so, is there much of a difference between (a) using poll with a ZMQ_FD and a normal fd, or (b) zmq_poll with a zmq socket and a normal fd?
[15:39] sustrik djc: no
[15:39] djc basically equivalent?
[15:39] sustrik just keep in mind that ZMQ_FD is edge triggered
[15:39] djc yeah :)
[15:39] djc and that's not an issue with zmq_poll because it does the extra ZMQ_EVENTS check, presumably?
[15:40] djc or s/check/loop/
[15:42] sustrik yes
[15:42] sustrik see zmq_poll implementation in src/zmq.cpp
[15:42] sustrik it uses ZMQ_EVENTS and ZMQ_FD
[15:42] djc okay, so it's basically the same as what I'm doing
[15:42] djc okay, nice to know :)
[15:42] djc btw, zeromq 2.1 is a very nice improvement over 2.0
[15:43] cremes what's the reasoning for ZMQ_RATE, ZMQ_HWM, etc using int64_t or uint64_t but ZMQ_LINGER (and 2 others) use int?
[15:44] djc (oh yeah, and I'm also looking forward a lot to all the sockopts being int, so that the API can be simpler than the current thing)
[15:47] pieterh cremes: zero-mq, zero-reasoning!
[15:48] pieterh these were added ad-hoc
[15:48] pieterh in 3.0 they're all int where they can be
[15:48] cremes pieterh: i had opened a bug on that some time ago... i just remembered
[15:48] pieterh anyhow, every language binding should wrap these IMO
[15:48] cremes sustrik closed it because he wanted to go from int64 -> int and not int -> int64
[15:48] pieterh so e.g. in czmq you have wrappers for all these
[15:49] cremes right, i was trying to make them consistent for the ruby binding
[15:49] cremes so that other folks wouldn't need to worry about it
[15:49] pieterh yeah, in the czmq code I have two ifdef blocks
[15:49] pieterh you're free to reuse that code if it helps
[15:50] cremes nope, got my issue fixed already... i exposed a problem with the ruby binding on 32-bit windows
[15:50] pieterh aight
[15:50] cremes and it was related to this
[15:51] pieterh sustrik: hey, I was kidding about the zero-reasoning bit... come back!
[15:52] pieterh mato: you there?
[15:57] mato pieterh: yes, but i need to go in like 5mins
[15:57] pieterh I think ephemeral ports is a great idea
[15:57] pieterh it should be easy to make, too
[15:58] pieterh I assume Sustrik will accept a patch but won't make this himself
[15:58] mato i can make a patch when i have time, it's not too hard
[15:58] mato the tricky bit is actually the getsockname()/other API
[15:58] mato since a socket can have many endpoints
[15:59] pieterh right, it can be the last one or somesuch
[15:59] pieterh it's just an int value, right?
[15:59] mato pieterh: no it can't
[15:59] mato if we make an API, we do it properly
[16:00] pieterh indeed
[16:00] pieterh so maybe it's worth just jotting down how the API should work first
[16:00] Seta00 looking forward to subports. my users already complain about copying libzmq.dll/so to a folder, I don't want to see their reactions when I tell them to open two ports
[16:00] pieterh yeah
[16:00] pieterh but that thread on subports was so *confused*
[16:01] mato i gotta go, cyl
[16:01] pieterh cyl mato
[16:01] pieterh zmq_push() { m=$(cat) && echo -e $(printf '\\x01\\x00\\x%02x\\x00%s' $((1 + ${#m})) "$m") | nc -q1 $@; }
[16:01] pieterh bash 1-liner from Daniel Lundin
[16:18] Bar "We'll generate random values, just like the real weather stations do."
[16:18] Bar lol
[16:23] mikko aha!
[16:23] mikko it got me again, setting HWM before connect
[16:27] djc mikko: what does that do?
[16:28] mikko the hwm has no effect unless you set it before connect/bind
[16:28] mikko i keep tripping on that
[16:43] pieterh mikko: if you keep tripping over it, it's bad design
[16:44] pieterh raise an issue, the setsockopt should return -1 if you call it after connect/bind
[16:44] pieterh (unless it does weird stuff like applying to some but not all connections... agh... )
[16:44] sustrik exactly
[16:44] pieterh bad design, if people keep getting it wrong
[16:45] pieterh this is like a basic rule of design, it should not be surprising
[16:46] sustrik what's the alternative?
[16:49] mikko sustrik: have a method for setting the sockopt for specific connection
[16:49] mikko not sure if that is too far from posix
[16:49] sustrik quite far imo
[16:49] sustrik it woudl require "connect IDs"
[16:49] sustrik and "bind IDs"
[16:50] mikko that is not all bad
[16:50] mikko would allow removing connection as well
[16:50] mikko currently you have to close socket and rebuild without one connection
[16:51] pieterh sustrik: what would you expect to happen, as a user, if you set HWM on a socket that's already connected...?
[16:51] sustrik no change
[16:52] pieterh nope, that's as an implementor
[16:52] pieterh since Mikko, a classic user, keeps tripping up on this, 'no change' is clearly not what he intends
[16:52] sustrik change to running sockets don't work well
[16:52] sustrik mailbox_t is nice example
[16:53] pieterh you're telling me, as often, about implementation details
[16:53] pieterh I'm talking about API design
[16:53] sustrik (the probleb with resizing buffers on the run)
[16:53] sustrik i am talking about how existing implementations work
[16:53] pieterh we have this kind of discussion often
[16:53] sustrik the rule of the thumb is "don't do that or you get burned"
[16:55] pieterh allow me to suggest that APIs should not burn their users
[16:55] pieterh and should properly report errors when users make mistakes
[16:56] sustrik sure, but keep inside POSIX
[16:56] pieterh what does that mean?
[16:56] pieterh please send me the URL for the "POSIX API"
[16:56] pieterh it is massive
[16:56] sustrik use interfaces as defined at
[16:57] pieterh I'd point out that the Open Group *develop* standards, meaning it's a process
[16:58] pieterh however I'm interested in what specific API standards you're following in this case
[16:58] pieterh since you often claim "POSIX" (which is a good goal IMO) it would be useful to be precise about what that means
[16:58] pieterh "silent burning of users who get it wrong" isn't in there IMO
[16:59] sustrik that's what happens in reality though
[16:59] pieterh and you accept this as inevitable?
[16:59] sustrik changing options on run-time is hard or maybe impossible to implement consistently
[16:59] sustrik depends on the option
[17:00] pieterh so with HWMs you can maintain different-sized queues for each active connection in a socket?
[17:00] pieterh but you cannot resize a queue nor copy one queue to another
[17:01] pieterh surely the HWM is just a number
[17:02] sustrik yes, that's the case
[17:02] pieterh if your queue is already larger than the HWM, setting it lower won't have immediate impact
[17:02] pieterh only when the queue grows again
[17:02] pieterh thus, setting the HWM on an empty queue can work trivially
[17:02] sustrik there are synchronisation problems
[17:03] sustrik you can never be sure that the queue is empty at the moment
[17:03] pieterh does that matter?
[17:03] pieterh further, it would be simpler than today
[17:03] sustrik dunno, maybe it's possible to do it
[17:03] sustrik but very complex
[17:03] sustrik you can try
[17:03] pieterh now you have to manage a separate HWM per queue
[17:04] pieterh well, "send me a patch" is a good joke but it doesn't work on me
[17:04] pieterh my role is limited here to challenging your (fairly frequent) assertions that painful APIs are inevitable due to implementation details :)
[17:05] pieterh ok, I owe you a beer next week for being rather unkind
[17:06] pieterh however it is always profitable to improve an area where people systematically make mistakes
[17:10] sustrik well, it's
[17:10] sustrik 1. hard to do
[17:10] sustrik OSes often get it wrong, which is a sign of a possible problem there
[17:10] sustrik 2. it's not clear how to expose it via POSIX api
[17:11] sustrik 3. the development of new features is severely hindered by the hacks in the codebase that cannot be removed because people are using them
[17:12] sustrik thus, adding any more hacks can easily kill the project
[17:12] sustrik what we need to do before doing this kind of thing is to grow the developer base
[17:13] sustrik once there are 10-20 people hacking on the core
[17:14] sustrik we can allow some more complexity in the codebase
[18:06] CIA-75 pyzmq: 03hugo shi 07master * ra8e6532 10/ examples/eventloop/ : added simple example that runs a tornado web server which can send zmq messages to ...
[18:34] CIA-75 pyzmq: 03Min RK 07master * r3556e77 10/ (zmq/core/socket.pyx zmq/tests/ Merge pull request #108 from minrk/testsockets. ...
[18:56] Bartzy I was just told at ##php that PHP is totally unsuitable for daemons and servers and stuff
[18:56] Bartzy ;(
[18:58] Bartzy Anyone here is using PHP as a 'server' ?
[18:59] Bartzy with 0MQ, of course...
[21:02] aspidites any best practices when trying to design a network architecture? or is it more of a trial and error thing?
[21:03] mikko aspidites: same best practices apply as outside zeromq
[21:04] mikko there are some zeromq specific established things as well
[21:04] mikko but if you explain your situation a bit people can advise better
[21:06] aspidites well i'm trying to write a server for a trading card game. i'm thinking about having deck objects created on the server and allowing those objects to be modified using a req/rep socket pair, simultaneously using pub/sub sockets to deliver/receive updates
[21:06] aspidites eg, player client can request to draw cards and receive a reply with the cards that were drawn, but other players or spectators could subscribe to deck events such as 'player 1 drew a card'
[21:08] mikko ok, is this over the internet?
[21:09] aspidites desktop
[21:09] aspidites server would be run as a daemon either on a remote machine, or on a 'host' machine, clients would be desktop applications that connected to said remote machines, more than likely locating them through a simple name server
[21:10] aspidites initially i'll just key in the server address as a command line option, so i'm not so worried about the name server right now
[21:11] mikko what i meant by over the internet is that are you exposing zeromq to internet?
[21:11] mikko there are still occasional asserts on invalid messages
[21:11] mikko but those usually get fixed pretty quickly as they are found
[21:12] aspidites i guess i still don't follow.
[21:13] aspidites there won't be (initially) any web based clients if that's what you mean
[21:13] mikko i mean public access
[21:13] mikko over the internet
[21:13] mikko are you exposing zeromq to public internet or is this lan application?
[21:14] aspidites mostly lan
[21:14] aspidites actually no. it would be public internet
[21:15] aspidites sorry i'm being confusing. users will most likely be spread accross domains, in separate cities, states, and more likely countries, not on the same router
[21:17] mikko yes
[21:17] mikko thats what i meant
[21:18] aspidites yeah. public internet. might need to slow down on drinking those energy drinks
[21:19] aspidites perhaps i can ask a simpler question: should i prefer the req/rep pattern when i need to guarantee delivery of messages, and the pub/sub pattern when clients may come and go without carrying about what messages are received?
[21:19] mikko yes, but most likely you want XREP/XREQ
[21:22] aspidites which is now router/dealer, right?
[21:23] mikko yes
[21:23] aspidites ok. so when would i use push/pull?
[21:23] mikko fire and forget
[21:25] aspidites ah ok
[21:26] aspidites i see the multithreading example, is there a similar multiprocessing example?
[21:30] aspidites nvm thanks though
[23:24] Bartzy Can I use fork with zeromq ?
[23:24] Bartzy It won't crash or something cause file descriptors get shared between processes ?