Saturday September 11, 2010

[Time] NameMessage
[11:44] icy I can connect a socket to many endpoints, what is the reason for not allowing one to disconnect from an endpoint again?
[12:00] icy also why do I have to loop over *all* sockets to see which ones got events after a zmq_poll(), I can't see how this would scale
[12:03] guido_g that's how poll works
[12:08] icy well that kinda nullifies a benefit that for example epoll() gives you
[12:09] icy where you don't have to loop around searching
[12:11] guido_g a) somewhere you have to go through the list of fds
[12:12] guido_g b) ømq sockets aggregate os level sockets (one ømq socket -> many os level sockets), so you have only a few ømq sockets
[12:12] guido_g c) ømq uses epoll or whatever is available on the given platform
[12:12] icy yea but due to not being able to disconnect from one endpoint, I might end up needing many zmq sockets
[12:13] guido_g i don't dare to ask why...
[12:13] icy it uses epoll but makes me loop over all sockets, epoll does not require me to do that
[12:13] guido_g epoll is not available on all platforms ømq is running on
[12:13] icy I know
[12:13] guido_g so?
[12:14] icy but many provide smarter facilities than poll()
[12:14] icy so you throw away a lot of benefits for a lowest common denominator
[12:15] guido_g so you think it would make sense to have ømq behave differently on differnt platforms?
[12:15] icy that is not what I am saying
[12:15] guido_g it is
[12:15] icy nope
[12:15] guido_g because if you use the _best_ way on a platform, you also have the platform specific semantics
[12:16] icy you can still abstract it in a general way
[12:16] icy see all event libraries like libev or libevent
[12:16] gebi icy: just for the record, epoll is NOT faster than poll in some situations, quite frankly it is 20% or more slower than poll in some situations
[12:17] icy I know, for small sets poll() is nice
[12:17] guido_g may be these abstraction would cost a lot on some platforms, so why punish people not running you favorite platform?
[12:18] icy they do not cost any considerable amount
[12:18] guido_g evidence by allegation?
[12:19] icy what evidence do you provide?
[12:19] guido_g ømq is built that way, and works
[12:21] icy well all I'm saying is I cannot see how it can scale to many zmq sockets because of all the looping. so clearly there must be a reason for that and that would that you would not need many zqm sockets as each can connect to many transport endpoints
[12:22] icy which brings me to my initial question why you can't disconnect from one again
[12:23] icy I might be missing something in how you are supposed to handle many volatile connections and am ready for enlightenment :)
[12:25] guido_g what kind of connections?
[12:25] guido_g you don't see connects at the ømq level
[12:28] icy hm I was thinking of a socket which connects to many IPs over tcp but I guess I just need to flip it around and make it the server and not client
[12:29] guido_g even then you would have only one ømq socket
[12:29] pieter_hintjens gebi: yes, i recall Zed Shaw's analysis, quite fun
[12:30] pieter_hintjens icy: you're IMO trying to apply existing TCP patterns to 0MQ
[12:30] icy yea but if the IPs are dynamic, the zmq socket would just get more and more endpoints attached to it
[12:30] icy pieter_hintjens: quite possible, I am used to that :)
[12:30] guido_g if you connect more and more sockets to a remote socket, then yes, the number of connections is growing
[12:30] pieter_hintjens icy: you need to think of 0MQ sockets as mini applications not just APIs
[12:31] pieter_hintjens they are intelligent and do things for you
[12:31] pieter_hintjens learning this is the key to using 0MQ right and getting it to scale and be useful for you
[12:31] pieter_hintjens there are some generic rules that apply to all 0MQ sockets
[12:32] pieter_hintjens and then there are specifics to each type, how they behave
[12:32] pieter_hintjens generically, client vs. server is important to get right
[12:32] pieter_hintjens stable nodes act better as servers (doing the bind), dynamic nodes as clients (doing the connect)
[12:32] icy yup makes sense, I think it will just take a bit of time to get the old patterns out of my head
[12:32] pieter_hintjens yup
[12:32] pieter_hintjens if it's any help, we all go through this
[12:32] pieter_hintjens did you read the user guide and try the examples?
[12:33] icy yes
[12:33] pieter_hintjens did you just read it, or try the code?
[12:33] icy I tried not all but some
[12:33] icy and read all
[12:33] pieter_hintjens well, it takes time too
[12:33] pieter_hintjens several days of slow reprogramming
[12:34] guido_g it *needs* time to sink in
[12:34] guido_g ømq can't be groked in a day, imho
[12:34] pieter_hintjens if you never did networking programming you'd have less trouble unlearning old patterns
[12:34] guido_g the things that makes it so good is what makes it so hard :)
[12:34] pieter_hintjens but you'd still have to learn the new stuff
[12:35] guido_g but who would that be?
[12:35] pieter_hintjens icy: briefly, you trust 0MQ sockets to do a lot for you
[12:35] pieter_hintjens there is a lot less of the command-and-control you need with traditional sockets
[12:36] icy well I understand how stuff works, not very complicated after all. I'm trying now to understand why the api is how it is. e.g. normally event libs let you specify callbacks that get called upon events (hence I was so surprised by the looping needed after zmq_poll)
[12:36] pieter_hintjens icy: the api aims to look like plain boring sockets
[12:37] pieter_hintjens the poll stuff is not very nice
[12:37] pieter_hintjens and we'll probably make it nicer for 3.0
[12:37] icy fair enough
[12:37] pieter_hintjens there is a moral when you make new technology
[12:37] pieter_hintjens if it looks new, people won't adopt it
[12:38] guido_g hmmm
[12:38] pieter_hintjens people = except pioneers who have a fetish for new technology just on principle
[12:38] pieter_hintjens and who will abandon it for the next new thing without hesitation
[12:39] guido_g i hmmed because the last 30 years have seen a lot of new things
[12:39] pieter_hintjens I read a lovely quote from the inventor of liquid salt nuclear reactors
[12:39] pieter_hintjens sure... i'm making a horrid generalization
[12:39] guido_g ack
[12:39] icy I'm not so sure if this can be applied so easily
[12:40] guido_g go on :)
[12:40] pieter_hintjens "Perhaps the moral to be drawn is that a technology that differs too much from an existing technology has not one hurdle to overcome--to demostrate its feasibility--but another even greater one--to convince influential individuals and organizations who are intellectually and emotionally attached to a different technology that they should adopt the new path"
[12:40] guido_g icy: it's a trade-off between usability and speed, mostly
[12:40] icy if it was true, no radically things would have been adopted in the past but there have been plenty :)
[12:40] pieter_hintjens ah, (a) there are always exceptions to a general rule and (b) there are ways to appear old yet introduce newness
[12:40] guido_g i've seen a messaging lib w/ callbacks, and it wasn't fun at all
[12:41] icy in my opinion anything providing a considerable advantage over something else has good chances of getting adopted
[12:41] icy provided it does not have disadvantages that outweight the advantages of course :P
[12:41] guido_g you remember the video-tape thingy?
[12:41] pieter_hintjens this is the engineer point of view but it ignores people emotional attachment to stuff they know
[12:41] pieter_hintjens *people's
[12:41] pieter_hintjens *peoples'
[12:42] pieter_hintjens sms text messages
[12:42] pieter_hintjens email
[12:42] icy guido_g: what about fire, electricity etc? :)
[12:42] pieter_hintjens icy: fire has callbacks?
[12:42] icy there are examples for both sides
[12:42] icy pieter_hintjens: uhm.. ;)
[12:42] pieter_hintjens :-)
[12:43] guido_g icy: but it doesn't work that way
[12:43] pieter_hintjens anyhow, this is why 0MQ very carefully emulates a poll function everyone hates but totally trusts and understands
[12:44] icy guido_g: just curious, why was that lib no fun?
[12:44] guido_g icy: wllm
[12:44] icy pieter_hintjens: I totally see your point
[12:45] icy wllm?
[12:45] pieter_hintjens icy: once there is a significant weight behind 0MQ (and we're getting there) we can be more ambitious about our own abstractions
[12:45] guido_g icy: from a small 3 letter company (HAL) :)
[12:46] guido_g icy:
[12:46] icy pieter_hintjens: wouldn't it be hard to make a radical api change once you got a considerable amount of applications using it? as a counter argument
[12:46] icy guido_g: will take a look, thx
[12:46] pieter_hintjens icy: it's about layers
[12:46] guido_g icy: no need to, really :)
[12:46] pieter_hintjens we'd never make radical api changes
[12:46] pieter_hintjens just add layers on top
[12:47] pieter_hintjens e.g. event driven frameworks that actually call the poll API
[12:47] pieter_hintjens though I suspect poll API itself will be redesigned before the 0MQ is totally stabilized
[12:47] guido_g so the developer can pick his poison depending on the use-case etc.
[12:52] pieter_hintjens yeah
[12:52] icy for the non blocking part I'd argue it makes it even easier given the atomicity of messages
[12:53] guido_g we had a lot of trouble w/ these callbacks
[12:54] guido_g one problem is that you are still in the library
[12:54] guido_g which leads to more or less well known constraints
[12:55] pieter_hintjens threading issues etc.
[12:55] guido_g i wouldn't go that route again
[12:55] pieter_hintjens the reactor pattern looks more sane
[12:55] guido_g pieter_hintjens: oh yes!
[12:55] guido_g to both
[12:55] pieter_hintjens what i'd love to make is a Libero/0MQ at some stage
[12:56] guido_g sorry, what would that be?
[12:56] pieter_hintjens well, something like that, anyhow: state machines where events are 0MQ messages
[12:56] pieter_hintjens you turn the loop inside out, the loop calls your code and feeds it incoming messages as events
[12:56] pieter_hintjens and your code goes through states
[12:56] guido_g makes sense
[12:57] pieter_hintjens it's a pattern that can be terribly useful but it can be hard to grok as well
[12:57] pieter_hintjens only worth doing for more complex 0MQ apps but I suspect we're heading that way
[12:57] pieter_hintjens devices that manage 3+ sockets
[12:57] guido_g sure
[12:59] icy yes this is what I was refering to. you give a socket a function to call upon receiving a message. this way you don't need to care about searching which sockets got a message after a poll
[13:00] icy zmq would do it for you and can therefor use the faster implementation possible for a given platform
[13:00] pieter_hintjens icy: you can trivially build this on top
[13:00] pieter_hintjens perhaps a fun thing to make
[13:00] guido_g icy: as i said, it wasn't nice to use in a real world app
[13:01] pieter_hintjens i can see several problems with it, though
[13:01] guido_g plus the problems comming from the product itself
[13:01] pieter_hintjens passing control to a function means you can't e.g. work with multiple sockets at once
[13:01] pieter_hintjens so you can't do stuff like devices that read from one, write to the other
[13:01] pieter_hintjens or you need to create and pass around increasingly complex contexts
[13:01] guido_g you can't even do a proper error handling :)
[13:02] guido_g which then slow you down even more
[13:02] pieter_hintjens but it could be fun to experiment with
[13:02] pieter_hintjens icy: take a look at the various reactors people like Matt Weinstein and Chuck Remes have made
[13:03] pieter_hintjens
[13:04] icy of course you can do proper error handling. maybe your experience was because of a poor product, not the methodology itself guido_g
[13:04] icy but yea to handle multiple sockets you need to pass around also some additional data
[13:05] guido_g callbacks are generally not the first choice, i did X11 and Motif and learned to avoid them
[13:05] guido_g then wllm, smame thing
[13:05] guido_g *same
[13:05] guido_g if you want it, you should do it in the app
[13:06] guido_g libs or frameworks can't know enough of the use-cases to provide a proper solution for every user
[13:06] guido_g this is why frameworks tend to become so big and bloated
[13:09] pieter_hintjens icy: what i've found with 0MQ is that theory is pretty useless :-)
[13:09] pieter_hintjens get your hands dirty with code and it becomes rapidly clear
[13:09] pieter_hintjens happily what you learn is the same no matter what language you use
[17:12] pieter_hintjens check out what Andrew Cholakian's just built with 0MQ:
[17:24] keffo um, push/pull is not pub/sub, or did i confuse things?
[17:25] keffo ah right
[19:03] iFire did this email go anywhere?
[19:04] iFire I mean is it feasible
[22:04] psino is there any way to tell a socket to stop connecting to an endpoint, or stop listening on an endpoint?
[22:19] psino and is there a way of gracefully shutting down a worker process that uses an incoming PULL socket?