Friday October 22, 2010

[Time] NameMessage
[07:30] mikko good morning
[07:39] pieterh mikko: hi :-)
[12:10] TheEffigy Hi all
[12:10] TheEffigy Does anyone know if it's possible to use ephermeral ports with zmq?
[12:12] guido_g what is the problem?
[12:13] TheEffigy well i'd like to set up a ZMQ_PAIR socket and bind it, but i don't really want to specify a port for it
[12:13] TheEffigy but i'm unsure if it is possible to know what port it would be listening on?
[12:13] guido_g no, you need to know the port in advance
[12:15] TheEffigy mm, figured as much
[12:16] TheEffigy is it actually possible to bind to the ephemeral range ports anyway?
[12:17] guido_g might depend on the os, on mine it is
[12:18] TheEffigy which is that?
[12:18] pieterh TheEffigy, what transport, tcp?
[12:18] TheEffigy yeah, tcp
[12:19] guido_g PAIR is tcp only if done over the network, isn't it?
[12:19] pieterh yes
[12:20] pieterh for outgoing only, i.e. not for binding
[12:20] guido_g but the ports thing is the same for epgm, which is udp
[12:21] pieterh TheEffigy, ephemeral ports, afaics, are just the mechanism TCP uses to create ports for outgoing connections
[12:21] pieterh this is what shows up if you do a netstat -a
[12:21] pieterh you cannot connect TO an ephemeral port, nor bind to it, that makes no sense
[12:22] pieterh what you want perhaps is a name resolution service
[12:22] pieterh so that client and server can both use a logical name rather than a network name + port...?
[12:23] TheEffigy hmm, yes perhaps that would do
[12:24] pieterh there's unfortunately no such thing for 0MQ, yet
[12:24] pieterh but it'll come
[12:24] TheEffigy i was just under the impression ephemeral ports in general were just the range of ports "reserved" but not in the commonly used range
[12:25] pieterh
[12:26] pieterh "Just as well-known and registered port numbers are used for server processes, ephemeral port numbers are for client processes only."
[12:27] guido_g great link, thx
[12:27] TheEffigy ahh, yes that does make more sense now that i think about it
[12:31] pieterh TheEffigy, another technique you can use, depending on the scenario, is dynamic port numbers
[12:32] pieterh e.g. server can find an available port by scanning (trying to bind from 4096 up until it works)
[12:32] pieterh and then send the endpoint info to the client via another socket
[12:33] pieterh but in general if you're using PAIR sockets, over TCP, 90% likely you're doing your architecture wrong
[12:35] TheEffigy yes perhaps
[12:35] TheEffigy i'll explain
[12:36] TheEffigy my understanding is that the pub/sub matching is based on the client side only
[12:37] TheEffigy i'm taking a brokered approach to the system that i'm building, which is bringing in data from many sources and streaming it to connected applications
[12:37] TheEffigy however they might not be interested in all the data, which can be easily categorised on a few levels
[12:38] TheEffigy i'd rather not send the data onwards if they're not interested as it would only be discarded anyway, and it is potentially quite a lot of data
[12:39] pieterh Ok, kind of a classic scenario
[12:39] pieterh so there are several options here
[12:40] pieterh a. wait until 0MQ does publisher-side filtering
[12:40] pieterh b. split data up over a set of static sockets
[12:40] pieterh c. create and manage one socket per subscriber
[12:40] TheEffigy c. is the approach i was going for
[12:41] pieterh first question would be, how many subscribers?
[12:41] TheEffigy no more than 50
[12:41] pieterh aight, so solution c would work
[12:41] pieterh i would do it as follows
[12:42] pieterh - req/rep socket for subscription requests
[12:42] TheEffigy yep - done that part already
[12:42] pieterh - one pub socket per subscriber
[12:43] pieterh - some kind of heartbeating to allow server to clean up
[12:43] pieterh - dynamic port selection using technique I already outlined
[12:43] pieterh - i.e. server looks for available port within some range, then tells the client this endpoint
[12:43] TheEffigy yep
[12:43] TheEffigy sounds good.
[12:43] pieterh that will work
[12:44] TheEffigy only question i have is basically what is the difference between using a pub socket instead of a pair?
[12:44] pieterh pair does not handle, for example, reconnections
[12:44] TheEffigy ah
[12:44] TheEffigy yeah, well that's enough to make it a deal breaker
[12:44] pieterh actually rather than pub, consider using PUSH/PULL or XREP/XREQ
[12:45] pieterh advantage of PUB is that you can connect more than one client to the server endpoint, transparently
[12:45] pieterh if you use PUSH or XREP, then if a second client connects to the same endpoint, messages will get load-balanced
[12:45] pieterh which is not what you want
[12:46] TheEffigy ah
[12:46] TheEffigy interesting.
[12:46] TheEffigy i'll have a read up on those
[12:46] pieterh so with PUB sockets you can do things like make subscriptions + endpoints available downstream
[12:46] pieterh to other subscribers
[12:46] pieterh without any risk that it'll affect existing ones
[12:46] TheEffigy excellent
[12:46] pieterh np
[12:47] TheEffigy thanks so much for pointing me in the right direction
[12:47] pieterh did you read the guide yet?
[12:48] pieterh i know this is not covered
[12:55] TheEffigy i did, but that was just before my holiday so i've forgotten some of the finer points. might be time for a quick refresher
[16:18] ptrb is zmq_socket threadsafe on the context?
[16:23] pieterh ptrb, what do you mean...?
[16:23] pieterh in 0MQ/2.0.x you cannot use a socket except in the thread you created it in
[16:24] pieterh this is explained in the Guide
[16:24] pieterh in 0MQ/2.1.x you can create a socket in one thread and use it in another, but not share it between two threads at once
[16:27] ptrb right, that's clear. I mean if I create a shared context, can multiple threads call zmq_socket(ctx, ...); on it without protection
[16:35] pieterh ptrb: yes, certainly
[16:35] pieterh you'll see a lot of that in the examples in the guide
[16:48] ptrb cool, great.