ZeroMq IRC Log

Saturday September 11, 2010

[Time] Name	Message
[11:44] icy	I can connect a socket to many endpoints, what is the reason for not allowing one to disconnect from an endpoint again?
[12:00] icy	also why do I have to loop over all sockets to see which ones got events after a zmq_poll(), I can't see how this would scale
[12:03] guido_g	that's how poll works
[12:08] icy	well that kinda nullifies a benefit that for example epoll() gives you
[12:09] icy	where you don't have to loop around searching
[12:11] guido_g	a) somewhere you have to go through the list of fds
[12:12] guido_g	b) Ã¸mq sockets aggregate os level sockets (one Ã¸mq socket -> many os level sockets), so you have only a few Ã¸mq sockets
[12:12] guido_g	c) Ã¸mq uses epoll or whatever is available on the given platform
[12:12] icy	yea but due to not being able to disconnect from one endpoint, I might end up needing many zmq sockets
[12:13] guido_g	i don't dare to ask why...
[12:13] icy	it uses epoll but makes me loop over all sockets, epoll does not require me to do that
[12:13] guido_g	epoll is not available on all platforms Ã¸mq is running on
[12:13] icy	I know
[12:13] guido_g	so?
[12:14] icy	but many provide smarter facilities than poll()
[12:14] icy	so you throw away a lot of benefits for a lowest common denominator
[12:15] guido_g	so you think it would make sense to have Ã¸mq behave differently on differnt platforms?
[12:15] icy	that is not what I am saying
[12:15] guido_g	it is
[12:15] icy	nope
[12:15] guido_g	because if you use the _best_ way on a platform, you also have the platform specific semantics
[12:16] icy	you can still abstract it in a general way
[12:16] icy	see all event libraries like libev or libevent
[12:16] gebi	icy: just for the record, epoll is NOT faster than poll in some situations, quite frankly it is 20% or more slower than poll in some situations
[12:17] icy	I know, for small sets poll() is nice
[12:17] guido_g	may be these abstraction would cost a lot on some platforms, so why punish people not running you favorite platform?
[12:18] icy	they do not cost any considerable amount
[12:18] guido_g	evidence by allegation?
[12:19] icy	what evidence do you provide?
[12:19] guido_g	Ã¸mq is built that way, and works
[12:21] icy	well all I'm saying is I cannot see how it can scale to many zmq sockets because of all the looping. so clearly there must be a reason for that and that would that you would not need many zqm sockets as each can connect to many transport endpoints
[12:22] icy	which brings me to my initial question why you can't disconnect from one again
[12:23] icy	I might be missing something in how you are supposed to handle many volatile connections and am ready for enlightenment :)
[12:25] guido_g	what kind of connections?
[12:25] guido_g	you don't see connects at the Ã¸mq level
[12:28] icy	hm I was thinking of a socket which connects to many IPs over tcp but I guess I just need to flip it around and make it the server and not client
[12:29] guido_g	even then you would have only one Ã¸mq socket
[12:29] pieter_hintjens	gebi: yes, i recall Zed Shaw's analysis, quite fun
[12:30] pieter_hintjens	icy: you're IMO trying to apply existing TCP patterns to 0MQ
[12:30] icy	yea but if the IPs are dynamic, the zmq socket would just get more and more endpoints attached to it
[12:30] icy	pieter_hintjens: quite possible, I am used to that :)
[12:30] guido_g	if you connect more and more sockets to a remote socket, then yes, the number of connections is growing
[12:30] pieter_hintjens	icy: you need to think of 0MQ sockets as mini applications not just APIs
[12:31] pieter_hintjens	they are intelligent and do things for you
[12:31] pieter_hintjens	learning this is the key to using 0MQ right and getting it to scale and be useful for you
[12:31] pieter_hintjens	there are some generic rules that apply to all 0MQ sockets
[12:32] pieter_hintjens	and then there are specifics to each type, how they behave
[12:32] pieter_hintjens	generically, client vs. server is important to get right
[12:32] pieter_hintjens	stable nodes act better as servers (doing the bind), dynamic nodes as clients (doing the connect)
[12:32] icy	yup makes sense, I think it will just take a bit of time to get the old patterns out of my head
[12:32] pieter_hintjens	yup
[12:32] pieter_hintjens	if it's any help, we all go through this
[12:32] pieter_hintjens	did you read the user guide and try the examples?
[12:33] icy	yes
[12:33] pieter_hintjens	did you just read it, or try the code?
[12:33] icy	I tried not all but some
[12:33] icy	and read all
[12:33] pieter_hintjens	well, it takes time too
[12:33] pieter_hintjens	several days of slow reprogramming
[12:34] guido_g	it needs time to sink in
[12:34] guido_g	Ã¸mq can't be groked in a day, imho
[12:34] pieter_hintjens	if you never did networking programming you'd have less trouble unlearning old patterns
[12:34] guido_g	the things that makes it so good is what makes it so hard :)
[12:34] pieter_hintjens	but you'd still have to learn the new stuff
[12:35] guido_g	but who would that be?
[12:35] pieter_hintjens	icy: briefly, you trust 0MQ sockets to do a lot for you
[12:35] pieter_hintjens	there is a lot less of the command-and-control you need with traditional sockets
[12:36] icy	well I understand how stuff works, not very complicated after all. I'm trying now to understand why the api is how it is. e.g. normally event libs let you specify callbacks that get called upon events (hence I was so surprised by the looping needed after zmq_poll)
[12:36] pieter_hintjens	icy: the api aims to look like plain boring sockets
[12:37] pieter_hintjens	the poll stuff is not very nice
[12:37] pieter_hintjens	and we'll probably make it nicer for 3.0
[12:37] icy	fair enough
[12:37] pieter_hintjens	there is a moral when you make new technology
[12:37] pieter_hintjens	if it looks new, people won't adopt it
[12:38] guido_g	hmmm
[12:38] pieter_hintjens	people = except pioneers who have a fetish for new technology just on principle
[12:38] pieter_hintjens	and who will abandon it for the next new thing without hesitation
[12:39] guido_g	i hmmed because the last 30 years have seen a lot of new things
[12:39] pieter_hintjens	I read a lovely quote from the inventor of liquid salt nuclear reactors
[12:39] pieter_hintjens	sure... i'm making a horrid generalization
[12:39] guido_g	ack
[12:39] icy	I'm not so sure if this can be applied so easily
[12:40] guido_g	go on :)
[12:40] pieter_hintjens	"Perhaps the moral to be drawn is that a technology that differs too much from an existing technology has not one hurdle to overcome--to demostrate its feasibility--but another even greater one--to convince influential individuals and organizations who are intellectually and emotionally attached to a different technology that they should adopt the new path"
[12:40] guido_g	icy: it's a trade-off between usability and speed, mostly
[12:40] icy	if it was true, no radically things would have been adopted in the past but there have been plenty :)
[12:40] pieter_hintjens	ah, (a) there are always exceptions to a general rule and (b) there are ways to appear old yet introduce newness
[12:40] guido_g	i've seen a messaging lib w/ callbacks, and it wasn't fun at all
[12:41] icy	in my opinion anything providing a considerable advantage over something else has good chances of getting adopted
[12:41] icy	provided it does not have disadvantages that outweight the advantages of course :P
[12:41] guido_g	you remember the video-tape thingy?
[12:41] pieter_hintjens	this is the engineer point of view but it ignores people emotional attachment to stuff they know
[12:41] pieter_hintjens	*people's
[12:41] pieter_hintjens	*peoples'
[12:42] pieter_hintjens	sms text messages
[12:42] pieter_hintjens	email
[12:42] icy	guido_g: what about fire, electricity etc? :)
[12:42] pieter_hintjens	icy: fire has callbacks?
[12:42] icy	there are examples for both sides
[12:42] icy	pieter_hintjens: uhm.. ;)
[12:42] pieter_hintjens	:-)
[12:43] guido_g	icy: but it doesn't work that way
[12:43] pieter_hintjens	anyhow, this is why 0MQ very carefully emulates a poll function everyone hates but totally trusts and understands
[12:44] icy	guido_g: just curious, why was that lib no fun?
[12:44] guido_g	icy: wllm
[12:44] icy	pieter_hintjens: I totally see your point
[12:45] icy	wllm?
[12:45] pieter_hintjens	icy: once there is a significant weight behind 0MQ (and we're getting there) we can be more ambitious about our own abstractions
[12:45] guido_g	icy: from a small 3 letter company (HAL) :)
[12:46] guido_g	icy: http://publib.boulder.ibm.com/infocenter/wllm/v2r0/index.jsp
[12:46] icy	pieter_hintjens: wouldn't it be hard to make a radical api change once you got a considerable amount of applications using it? as a counter argument
[12:46] icy	guido_g: will take a look, thx
[12:46] pieter_hintjens	icy: it's about layers
[12:46] guido_g	icy: no need to, really :)
[12:46] pieter_hintjens	we'd never make radical api changes
[12:46] pieter_hintjens	just add layers on top
[12:47] pieter_hintjens	e.g. event driven frameworks that actually call the poll API
[12:47] pieter_hintjens	though I suspect poll API itself will be redesigned before the 0MQ is totally stabilized
[12:47] guido_g	so the developer can pick his poison depending on the use-case etc.
[12:52] pieter_hintjens	yeah
[12:52] icy	for the non blocking part I'd argue it makes it even easier given the atomicity of messages
[12:53] guido_g	we had a lot of trouble w/ these callbacks
[12:54] guido_g	one problem is that you are still in the library
[12:54] guido_g	which leads to more or less well known constraints
[12:55] pieter_hintjens	threading issues etc.
[12:55] guido_g	i wouldn't go that route again
[12:55] pieter_hintjens	the reactor pattern looks more sane
[12:55] guido_g	pieter_hintjens: oh yes!
[12:55] guido_g	to both
[12:55] pieter_hintjens	what i'd love to make is a Libero/0MQ at some stage
[12:56] guido_g	sorry, what would that be?
[12:56] pieter_hintjens	well, something like that, anyhow: state machines where events are 0MQ messages
[12:56] pieter_hintjens	you turn the loop inside out, the loop calls your code and feeds it incoming messages as events
[12:56] pieter_hintjens	and your code goes through states
[12:56] guido_g	makes sense
[12:57] pieter_hintjens	it's a pattern that can be terribly useful but it can be hard to grok as well
[12:57] pieter_hintjens	only worth doing for more complex 0MQ apps but I suspect we're heading that way
[12:57] pieter_hintjens	devices that manage 3+ sockets
[12:57] guido_g	sure
[12:59] icy	yes this is what I was refering to. you give a socket a function to call upon receiving a message. this way you don't need to care about searching which sockets got a message after a poll
[13:00] icy	zmq would do it for you and can therefor use the faster implementation possible for a given platform
[13:00] pieter_hintjens	icy: you can trivially build this on top
[13:00] pieter_hintjens	perhaps a fun thing to make
[13:00] guido_g	icy: as i said, it wasn't nice to use in a real world app
[13:01] pieter_hintjens	i can see several problems with it, though
[13:01] guido_g	plus the problems comming from the product itself
[13:01] pieter_hintjens	passing control to a function means you can't e.g. work with multiple sockets at once
[13:01] pieter_hintjens	so you can't do stuff like devices that read from one, write to the other
[13:01] pieter_hintjens	or you need to create and pass around increasingly complex contexts
[13:01] guido_g	you can't even do a proper error handling :)
[13:02] guido_g	which then slow you down even more
[13:02] pieter_hintjens	but it could be fun to experiment with
[13:02] pieter_hintjens	icy: take a look at the various reactors people like Matt Weinstein and Chuck Remes have made
[13:03] pieter_hintjens	http://www.zeromq.org/docs:labs
[13:04] icy	of course you can do proper error handling. maybe your experience was because of a poor product, not the methodology itself guido_g
[13:04] icy	but yea to handle multiple sockets you need to pass around also some additional data
[13:05] guido_g	callbacks are generally not the first choice, i did X11 and Motif and learned to avoid them
[13:05] guido_g	then wllm, smame thing
[13:05] guido_g	*same
[13:05] guido_g	if you want it, you should do it in the app
[13:06] guido_g	libs or frameworks can't know enough of the use-cases to provide a proper solution for every user
[13:06] guido_g	this is why frameworks tend to become so big and bloated
[13:09] pieter_hintjens	icy: what i've found with 0MQ is that theory is pretty useless :-)
[13:09] pieter_hintjens	get your hands dirty with code and it becomes rapidly clear
[13:09] pieter_hintjens	happily what you learn is the same no matter what language you use
[17:12] pieter_hintjens	check out what Andrew Cholakian's just built with 0MQ: http://blog.andrewvc.com/learn-zeromq-by-building-an-http-load-tester
[17:24] keffo	um, push/pull is not pub/sub, or did i confuse things?
[17:25] keffo	ah right
[19:03] iFire	did this email go anywhere? http://lists.zeromq.org/pipermail/zeromq-dev/2010-July/004674.html
[19:04] iFire	I mean is it feasible
[22:04] psino	is there any way to tell a socket to stop connecting to an endpoint, or stop listening on an endpoint?
[22:19] psino	and is there a way of gracefully shutting down a worker process that uses an incoming PULL socket?