ZeroMq IRC Log

Wednesday May 4, 2011

[Time] Name	Message
[10:38] pieterh	Bartzy\|work: depends...
[10:38] pieterh	what's up?
[10:39] pieterh	go for it
[10:40] pieterh	ok
[10:41] pieterh	have you read the Guide?
[10:41] CIA-75	libzmq: 03Tore Halvorsen 07master * r1012021 10/ src/xrep.cpp : Accessing an iterator after it is accessed is not valid. ...
[10:43] pieterh	it doesn't really matter
[10:43] CIA-75	libzmq: 03Martin Sustrik 07master * r3ef7c74 10/ AUTHORS : Tore Halvorsen added to the AUTHORS file ...
[10:43] pieterh	whatever is easier for you
[10:43] pieterh	obviously, yes
[10:43] pieterh	there are a number of patterns you can use for this
[10:44] pieterh	check the LRU queue patterns, for a start
[10:44] pieterh	you can make the workers each a single process that is single threaded
[10:44] pieterh	you can make the workers threads in a single multithreaded process
[10:45] pieterh	read the LRU queue pattern & description, plz
[10:45] pieterh	search for lruqueue if in doubt
[10:45] mikko	Bartzy\|work: why do you use gimp?
[10:45] mikko	just out of curiosity
[10:45] mikko	Bartzy\|work: imagemagick tends to be more geared towards headless processing
[10:45] pieterh	Bartzy\|work: most people would probably use imagemagick except it's harder to spell
[10:46] mikko	Bartzy\|work: limited how?
[10:46] mato	ImageMagick is also sloooow
[10:46] mikko	mato: graphicsmagick then
[10:46] mikko	Bartzy\|work: pretty much
[10:47] mikko	and probably more than in gimp
[10:47] mikko	last time my trial with gimp ended to not having proper CMYK support
[10:47] mikko	Bartzy\|work: what do you do in the gimp scripts?
[10:48] mikko	what kind of effects?
[10:48] mikko	programmatically?
[10:48] pieterh	Bartzy\|work: I'd suggest initially making one process per gimp server, single threaded, and use an lruqueue device to connect workers to clients
[10:49] mikko	cool
[10:50] sustrik	hi mikko
[10:50] mikko	hi
[10:50] sustrik	any chance to check patches at the build system?
[10:50] mikko	sustrik: sure
[10:50] mikko	sustrik: which ones?
[10:50] pieterh	Bartzy\|work: nope, all messaging is just 0MQ done poorly
[10:50] sustrik	i've tried to fix the issues by giess
[10:50] sustrik	guess
[10:51] sustrik	but obviously that does not work :)
[10:51] pieterh	Bartzy\|work: well, I'm assuming you haven't actually tried 0MQ
[10:51] sustrik	mikko: well, the sun studio and icc
[10:53] pieterh	Bartzy\|work: it's not about coolness, if you haven't actually written a couple of 0MQ apps, you can't understand it
[10:53] pieterh	so my advice is to go off, make a simple workload distribution architecture, in PHP
[10:53] pieterh	it'll take you a day, maybe
[10:53] pieterh	if you have questions, of course ask for help
[10:53] pieterh	but there are full, complete PHP examples for this
[10:54] pieterh	sigh
[10:54] pieterh	:)
[10:54] pieterh	start by learning 0MQ, then when you know what you want to make, and how, decide how to really make it
[10:55] pieterh	it's pointless to debate "what language" when you don't even know what architecture
[10:55] pieterh	"ah, but do you think I should use vim or emacs?"
[10:55] pieterh	not relevant
[10:55] pieterh	first learn some basic messaging patterns, then decide how to make them work for you
[10:56] pieterh	it's wise to make a proof of concept as rapidly as possible and then throw it away and start again
[10:56] pieterh	does this help?
[10:57] djc	Bartzy\|work: the point is that ZMQ is very good at having something built in a day
[10:57] djc	instead of in a week
[10:57] djc	so researching it is less valuable than for some other technologies
[10:59] djc	the same as what you do now
[10:59] pieterh	Bartzy\|work: the only research that counts is actually programming
[10:59] pieterh	asking others to make your technical choices for you isn't profitable
[10:59] pieterh	0MQ makes it really easy to go off and learn by doing, so please do so...
[11:00] mikko	Bartzy\|work: i've been talking with ian barber about php job distribution system before
[11:00] mikko	making use of php-fpm for process management
[11:01] mikko	Bartzy\|work: would not be very hard
[11:02] mikko	but lack of time
[11:02] ianbarber	it's a decent model, works fine. The tricky thing with whatever you use is maintaining the worker processes existence, the basic messaging is straightforward, and FPM simplifies that.
[11:03] ianbarber	otherwise that kind of work distribution is just basic push messaging
[11:05] djc	Bartzy\|work: read more of the guide, then just spend a day writing a proof of concept
[11:06] sustrik	mikko: i mean, can i log on the build box
[11:06] sustrik	and try to fix the sun studio and icc problems there?
[11:08] mikko	sustrik: it's behind a nat at the moment without public ip address
[11:08] mikko	sustrik: i can add you user to a jump host
[11:08] sustrik	hm, alternatively, can i send you patches to test?
[11:08] mikko	i think i got a box connected to the internet with sun studio
[11:08] mikko	if you want i can take a look in a minute
[11:08] sustrik	great, thanks
[11:10] mikko	https://build.zero.mq/view/libzmq/
[11:10] mikko	the builds succeed at the moment
[11:11] mikko	apart from clang which seems to be a compiler thing on amd64 box
[11:11] sustrik	hm, strange
[11:11] sustrik	yesterday evening there were errors
[11:11] sustrik	anyway, if everything works no, no need to panic
[11:11] sustrik	now*
[11:13] mikko	Minor patch to keep ICC compiler happy (commit: 5e329ba7cac8a52fbbd2c347064c2d9355009022) (detail)
[11:13] mikko	Deallocation functions in zmq.h and msg_t class are consistent. (commit: ceb5e1a0734b0c73bd7f74ec5094ae6ad4f9dfc4) (detail)
[11:13] mikko	these are the changes in latest build
[11:15] sustrik	yep, those are my guess patches
[11:15] sustrik	i thought they were already used in the previous build
[11:15] sustrik	it seems they weren't
[11:15] sustrik	so everyting is ok now
[11:20] mikko	cool
[13:58] pieterh	Bartzy\|work: yup
[13:58] pieterh	so you finished your prototype?
[13:58] pieterh	:)
[13:59] pieterh	I'd start with the very simplest architecture that I could make to test my concept
[13:59] pieterh	in that case it should take you only an hour or two
[14:00] pieterh	if you are asking my advice on how to make something, don't argue with the advice I give you :)
[14:00] pieterh	make a minimal working model
[14:00] pieterh	then improve it to solve specific problems you see with it
[14:00] pieterh	otherwise, you will spend time solving problems that may be irrelevant
[14:01] pieterh	such as... 1 process vs. N processes
[14:01] pieterh	ok, so this is what you want to build...
[14:01] pieterh	well...
[14:02] pieterh	sigh... if you ask and try to answer too many questions at once, you won't get anywhere quickly
[14:02] pieterh	please do the following:
[14:03] pieterh	- take the lruqueue example
[14:03] pieterh	- make a PHP worker that executes request by request, starting GIMP in a shell each time
[14:03] pieterh	- make a test client, in PHP that can drive this worker
[14:03] pieterh	- start multiple workers in parallel
[14:03] pieterh	- see how it works
[14:03] pieterh	... OK?
[14:03] pieterh	when you have that working, let's continue the discussion
[14:04] pieterh	it is not relevant, to this discussion, how you talk to GIMOP
[14:04] pieterh	GIMP
[14:04] pieterh	not at least until GIMP has 0MQ built into it
[14:04] pieterh	yes
[14:05] pieterh	it's pointless to ask me questions about how to use ZeroMQ when you have no experience actually using it
[14:05] pieterh	and when there is a 150-page explanation with full examples...
[14:05] pieterh	:) guaranteed you will ask different questions
[14:05] guido_g	omg
[14:05] pieterh	guido_g: help please
[14:05] pieterh	it's like asking about sex when you've never tried it
[14:05] guido_g	pieterh: there is nothing one can do
[14:06] pieterh	"is it true I need to get really drunk before hand?"
[14:06] pieterh	"or would it be better to have a long sauna?"
[14:07] guido_g	pieterh: people come here (or to other channels) and demand answers
[14:07] pieterh	Bartzy\|work: I don't want to be critical, it's good that you think ahead, but don't over-think it
[14:07] guido_g	of course w/o even the smalles achievements on their side
[14:07] pieterh	guido_g: luckily we're a caring and helpful community
[14:08] pieterh	but indeed it could be worth raising the barrier a little...
[14:08] guido_g	yes, this does help
[14:09] guido_g	it helped back in use-net times and it did the last years in an irc channel
[14:09] pieterh	Bartzy\|work: the best practice is, and I quote Zed Shaw on this, "programming, motherf***r"
[14:10] pieterh	if you come here with questions that actually refer to the context we all share, it's much easier
[14:10] guido_g	Bartzy\|work: if you can't walk, don't even try to run
[14:11] pieterh	good deal
[14:12] pieterh	If you still see the problem the same way after using 0MQ, you get your money back
[14:12] guido_g	oh pieterh while this case is close, there is another question wrt the brussels
[14:12] pieterh	tell me
[14:13] guido_g	did you made a decision on weather? going to select the clothes...
[14:13] pieterh	I've asked for excellent weather
[14:13] guido_g	great
[14:13] pieterh	but there's a long queue so my request might not get processed in time
[14:13] pieterh	it will be warm and sunny
[14:13] pieterh	but perhaps only in August or so...
[14:14] guido_g	hehe
[14:14] pieterh	they're not using 0MQ, obviously...
[14:14] guido_g	luckly it's a very short walk from and to the hotel
[14:14] guido_g	Ã¸mq might have been made in heaven, but it's not used there
[14:14] pieterh	lol :)
[14:15] guido_g	is there an idea where the warm-up meeting on monday will be?
[14:16] djc	what kind of event is there?
[14:16] guido_g	http://www.zeromq.org/event:brussels-2011-05-10
[14:20] djc	nice
[14:20] djc	hmm, too bad, I probably won't be able to make it this time
[14:21] mato	pieterh: can you make a short url for the brussels conf page?
[14:21] pieterh	mato: it's been there for ages: http://zero.mq/bxl
[14:21] mato	ah
[14:21] pieterh	I put that in every email
[14:21] pieterh	:-(
[14:21] pieterh	let me put that on the page as well
[14:21] guido_g	it's too short :)
[14:21] pieterh	BTW anyone can make a short URL at zero.mq/go
[14:22] pieterh	djc: thanks :)
[14:22] mato	ok, that's better
[14:23] mato	pieterh: happy with that topic?
[14:24] pieterh	mato: you're the channel admin?
[14:24] mato	pieterh: one of them, yes
[14:24] pieterh	yes, very happy with this, was going to ask for it before but forgot
[14:24] pieterh	thanks!
[14:24] mato	np
[15:04] cremes	mikko: ping; question on compiling under mingw
[15:05] cremes	mikko: i run ./configure; make; make install and let it figure out the defaults
[15:05] cremes	mikko: the makefile produces libzmq.dll.a and libzmq.la but i do not see any libzmq.dll
[15:05] cremes	am i doing something wrong? do i need to pass another switch to configure?
[15:06] mato	cremes: look in src/.libs
[15:06] mato	libtool will hide the dll there
[15:07] cremes	ah ha!
[15:08] cremes	any particular reason 'make install' doesn't do the right thing here?
[15:08] mato	yes, what would you have it do on windows? :-)
[15:10] cremes	copy libzmq.dll to the same place it copies libzmq.dll.a and libzmq.la
[15:10] mato	there is no such place on win32
[15:11] cremes	well, under mingw it's pretty standard to install to /c/user/local/lib
[15:11] cremes	that's at least a good "known" place to copy it from to other locations
[15:11] cremes	as required by your windows apps
[15:11] Seta00	windows for /usr/lib is C:/Windows/System32 :P
[15:12] cremes	instead of hiding it in src/.libs (i swear i never would have found it there)
[15:12] mato	I guess it could create a 'bin' directory on mingw and stick things there
[15:12] mato	but AFAIK there is no real standard install path
[15:12] cremes	(i should have run 'find /c -name libzmq.dll"
[15:12] mato	other than %WINDIR%\system32 and you don't want that :-)
[15:14] cremes	got my first 0mq program running on windows! thanks for your help!
[15:36] djc	so, is there much of a difference between (a) using poll with a ZMQ_FD and a normal fd, or (b) zmq_poll with a zmq socket and a normal fd?
[15:39] sustrik	djc: no
[15:39] djc	basically equivalent?
[15:39] sustrik	just keep in mind that ZMQ_FD is edge triggered
[15:39] djc	yeah :)
[15:39] djc	and that's not an issue with zmq_poll because it does the extra ZMQ_EVENTS check, presumably?
[15:40] djc	or s/check/loop/
[15:42] sustrik	yes
[15:42] sustrik	see zmq_poll implementation in src/zmq.cpp
[15:42] sustrik	it uses ZMQ_EVENTS and ZMQ_FD
[15:42] djc	okay, so it's basically the same as what I'm doing
[15:42] djc	okay, nice to know :)
[15:42] djc	btw, zeromq 2.1 is a very nice improvement over 2.0
[15:43] cremes	what's the reasoning for ZMQ_RATE, ZMQ_HWM, etc using int64_t or uint64_t but ZMQ_LINGER (and 2 others) use int?
[15:44] djc	(oh yeah, and I'm also looking forward a lot to all the sockopts being int, so that the API can be simpler than the current thing)
[15:47] pieterh	cremes: zero-mq, zero-reasoning!
[15:48] pieterh	these were added ad-hoc
[15:48] pieterh	in 3.0 they're all int where they can be
[15:48] cremes	pieterh: i had opened a bug on that some time ago... i just remembered
[15:48] pieterh	anyhow, every language binding should wrap these IMO
[15:48] cremes	sustrik closed it because he wanted to go from int64 -> int and not int -> int64
[15:48] pieterh	so e.g. in czmq you have wrappers for all these
[15:49] cremes	right, i was trying to make them consistent for the ruby binding
[15:49] cremes	so that other folks wouldn't need to worry about it
[15:49] pieterh	yeah, in the czmq code I have two ifdef blocks
[15:49] pieterh	you're free to reuse that code if it helps
[15:50] cremes	nope, got my issue fixed already... i exposed a problem with the ruby binding on 32-bit windows
[15:50] pieterh	aight
[15:50] cremes	and it was related to this
[15:51] pieterh	sustrik: hey, I was kidding about the zero-reasoning bit... come back!
[15:52] pieterh	mato: you there?
[15:57] mato	pieterh: yes, but i need to go in like 5mins
[15:57] pieterh	I think ephemeral ports is a great idea
[15:57] pieterh	it should be easy to make, too
[15:58] pieterh	I assume Sustrik will accept a patch but won't make this himself
[15:58] mato	i can make a patch when i have time, it's not too hard
[15:58] mato	the tricky bit is actually the getsockname()/other API
[15:58] mato	since a socket can have many endpoints
[15:59] pieterh	right, it can be the last one or somesuch
[15:59] pieterh	it's just an int value, right?
[15:59] mato	pieterh: no it can't
[15:59] mato	if we make an API, we do it properly
[16:00] pieterh	indeed
[16:00] pieterh	so maybe it's worth just jotting down how the API should work first
[16:00] Seta00	looking forward to subports. my users already complain about copying libzmq.dll/so to a folder, I don't want to see their reactions when I tell them to open two ports
[16:00] pieterh	yeah
[16:00] pieterh	but that thread on subports was so confused
[16:01] mato	i gotta go, cyl
[16:01] pieterh	cyl mato
[16:01] pieterh	zmq_push() { m=$(cat) && echo -e $(printf '\\x01\\x00\\x%02x\\x00%s' $((1 + ${#m})) "$m") \| nc -q1 $@; }
[16:01] pieterh	bash 1-liner from Daniel Lundin
[16:18] Bar	"We'll generate random values, just like the real weather stations do."
[16:18] Bar	lol
[16:23] mikko	aha!
[16:23] mikko	it got me again, setting HWM before connect
[16:27] djc	mikko: what does that do?
[16:28] mikko	the hwm has no effect unless you set it before connect/bind
[16:28] mikko	i keep tripping on that
[16:43] pieterh	mikko: if you keep tripping over it, it's bad design
[16:44] pieterh	raise an issue, the setsockopt should return -1 if you call it after connect/bind
[16:44] pieterh	(unless it does weird stuff like applying to some but not all connections... agh... )
[16:44] sustrik	exactly
[16:44] pieterh	bad design, if people keep getting it wrong
[16:45] pieterh	this is like a basic rule of design, it should not be surprising
[16:46] sustrik	what's the alternative?
[16:49] mikko	sustrik: have a method for setting the sockopt for specific connection
[16:49] mikko	not sure if that is too far from posix
[16:49] sustrik	quite far imo
[16:49] sustrik	it woudl require "connect IDs"
[16:49] sustrik	and "bind IDs"
[16:50] mikko	that is not all bad
[16:50] mikko	would allow removing connection as well
[16:50] mikko	currently you have to close socket and rebuild without one connection
[16:51] pieterh	sustrik: what would you expect to happen, as a user, if you set HWM on a socket that's already connected...?
[16:51] sustrik	no change
[16:52] pieterh	nope, that's as an implementor
[16:52] pieterh	since Mikko, a classic user, keeps tripping up on this, 'no change' is clearly not what he intends
[16:52] sustrik	change to running sockets don't work well
[16:52] sustrik	mailbox_t is nice example
[16:53] pieterh	you're telling me, as often, about implementation details
[16:53] pieterh	I'm talking about API design
[16:53] sustrik	(the probleb with resizing buffers on the run)
[16:53] sustrik	i am talking about how existing implementations work
[16:53] pieterh	we have this kind of discussion often
[16:53] sustrik	the rule of the thumb is "don't do that or you get burned"
[16:55] pieterh	allow me to suggest that APIs should not burn their users
[16:55] pieterh	and should properly report errors when users make mistakes
[16:56] sustrik	sure, but keep inside POSIX
[16:56] pieterh	what does that mean?
[16:56] pieterh	please send me the URL for the "POSIX API"
[16:56] pieterh	it is massive
[16:56] sustrik	use interfaces as defined at opengroup.org
[16:57] pieterh	I'd point out that the Open Group develop standards, meaning it's a process
[16:58] pieterh	however I'm interested in what specific API standards you're following in this case
[16:58] pieterh	since you often claim "POSIX" (which is a good goal IMO) it would be useful to be precise about what that means
[16:58] pieterh	"silent burning of users who get it wrong" isn't in there IMO
[16:59] sustrik	that's what happens in reality though
[16:59] pieterh	and you accept this as inevitable?
[16:59] sustrik	changing options on run-time is hard or maybe impossible to implement consistently
[16:59] sustrik	depends on the option
[17:00] pieterh	so with HWMs you can maintain different-sized queues for each active connection in a socket?
[17:00] pieterh	but you cannot resize a queue nor copy one queue to another
[17:01] pieterh	surely the HWM is just a number
[17:02] sustrik	yes, that's the case
[17:02] pieterh	if your queue is already larger than the HWM, setting it lower won't have immediate impact
[17:02] pieterh	only when the queue grows again
[17:02] pieterh	thus, setting the HWM on an empty queue can work trivially
[17:02] sustrik	there are synchronisation problems
[17:03] sustrik	you can never be sure that the queue is empty at the moment
[17:03] pieterh	does that matter?
[17:03] pieterh	further, it would be simpler than today
[17:03] sustrik	dunno, maybe it's possible to do it
[17:03] sustrik	but very complex
[17:03] sustrik	you can try
[17:03] pieterh	now you have to manage a separate HWM per queue
[17:04] pieterh	well, "send me a patch" is a good joke but it doesn't work on me
[17:04] pieterh	my role is limited here to challenging your (fairly frequent) assertions that painful APIs are inevitable due to implementation details :)
[17:05] pieterh	ok, I owe you a beer next week for being rather unkind
[17:06] pieterh	however it is always profitable to improve an area where people systematically make mistakes
[17:10] sustrik	well, it's
[17:10] sustrik	1. hard to do
[17:10] sustrik	OSes often get it wrong, which is a sign of a possible problem there
[17:10] sustrik	2. it's not clear how to expose it via POSIX api
[17:11] sustrik	3. the development of new features is severely hindered by the hacks in the codebase that cannot be removed because people are using them
[17:12] sustrik	thus, adding any more hacks can easily kill the project
[17:12] sustrik	what we need to do before doing this kind of thing is to grow the developer base
[17:13] sustrik	once there are 10-20 people hacking on the core
[17:14] sustrik	we can allow some more complexity in the codebase
[18:06] CIA-75	pyzmq: 03hugo shi 07master * ra8e6532 10/ examples/eventloop/web.py : added simple example that runs a tornado web server which can send zmq messages to echostream.py ...
[18:34] CIA-75	pyzmq: 03Min RK 07master * r3556e77 10/ (zmq/core/socket.pyx zmq/tests/test_device.py): Merge pull request #108 from minrk/testsockets. ...
[18:56] Bartzy	I was just told at ##php that PHP is totally unsuitable for daemons and servers and stuff
[18:56] Bartzy	;(
[18:58] Bartzy	Anyone here is using PHP as a 'server' ?
[18:59] Bartzy	with 0MQ, of course...
[21:02] aspidites	any best practices when trying to design a network architecture? or is it more of a trial and error thing?
[21:03] mikko	aspidites: same best practices apply as outside zeromq
[21:04] mikko	there are some zeromq specific established things as well
[21:04] mikko	but if you explain your situation a bit people can advise better
[21:06] aspidites	well i'm trying to write a server for a trading card game. i'm thinking about having deck objects created on the server and allowing those objects to be modified using a req/rep socket pair, simultaneously using pub/sub sockets to deliver/receive updates
[21:06] aspidites	eg, player client can request to draw cards and receive a reply with the cards that were drawn, but other players or spectators could subscribe to deck events such as 'player 1 drew a card'
[21:08] mikko	ok, is this over the internet?
[21:09] aspidites	desktop
[21:09] aspidites	server would be run as a daemon either on a remote machine, or on a 'host' machine, clients would be desktop applications that connected to said remote machines, more than likely locating them through a simple name server
[21:10] aspidites	initially i'll just key in the server address as a command line option, so i'm not so worried about the name server right now
[21:11] mikko	what i meant by over the internet is that are you exposing zeromq to internet?
[21:11] mikko	there are still occasional asserts on invalid messages
[21:11] mikko	but those usually get fixed pretty quickly as they are found
[21:12] aspidites	i guess i still don't follow.
[21:13] aspidites	there won't be (initially) any web based clients if that's what you mean
[21:13] mikko	i mean public access
[21:13] mikko	over the internet
[21:13] mikko	are you exposing zeromq to public internet or is this lan application?
[21:14] aspidites	mostly lan
[21:14] aspidites	actually no. it would be public internet
[21:15] aspidites	sorry i'm being confusing. users will most likely be spread accross domains, in separate cities, states, and more likely countries, not on the same router
[21:17] mikko	yes
[21:17] mikko	thats what i meant
[21:18] aspidites	yeah. public internet. might need to slow down on drinking those energy drinks
[21:19] aspidites	perhaps i can ask a simpler question: should i prefer the req/rep pattern when i need to guarantee delivery of messages, and the pub/sub pattern when clients may come and go without carrying about what messages are received?
[21:19] mikko	yes, but most likely you want XREP/XREQ
[21:22] aspidites	which is now router/dealer, right?
[21:23] mikko	yes
[21:23] aspidites	ok. so when would i use push/pull?
[21:23] mikko	fire and forget
[21:25] aspidites	ah ok
[21:26] aspidites	i see the multithreading example, is there a similar multiprocessing example?
[21:30] aspidites	nvm thanks though
[23:24] Bartzy	Can I use fork with zeromq ?
[23:24] Bartzy	It won't crash or something cause file descriptors get shared between processes ?