ZeroMq IRC Log

Friday October 22, 2010

[Time] Name	Message
[07:30] mikko	good morning
[07:39] pieterh	mikko: hi :-)
[12:10] TheEffigy	Hi all
[12:10] TheEffigy	Does anyone know if it's possible to use ephermeral ports with zmq?
[12:12] guido_g	what is the problem?
[12:13] TheEffigy	well i'd like to set up a ZMQ_PAIR socket and bind it, but i don't really want to specify a port for it
[12:13] TheEffigy	but i'm unsure if it is possible to know what port it would be listening on?
[12:13] guido_g	no, you need to know the port in advance
[12:15] TheEffigy	mm, figured as much
[12:16] TheEffigy	is it actually possible to bind to the ephemeral range ports anyway?
[12:17] guido_g	might depend on the os, on mine it is
[12:18] TheEffigy	which is that?
[12:18] pieterh	TheEffigy, what transport, tcp?
[12:18] TheEffigy	yeah, tcp
[12:19] guido_g	PAIR is tcp only if done over the network, isn't it?
[12:19] pieterh	yes
[12:20] pieterh	for outgoing only, i.e. not for binding
[12:20] guido_g	but the ports thing is the same for epgm, which is udp
[12:21] pieterh	TheEffigy, ephemeral ports, afaics, are just the mechanism TCP uses to create ports for outgoing connections
[12:21] pieterh	this is what shows up if you do a netstat -a
[12:21] pieterh	you cannot connect TO an ephemeral port, nor bind to it, that makes no sense
[12:22] pieterh	what you want perhaps is a name resolution service
[12:22] pieterh	so that client and server can both use a logical name rather than a network name + port...?
[12:23] TheEffigy	hmm, yes perhaps that would do
[12:24] pieterh	there's unfortunately no such thing for 0MQ, yet
[12:24] pieterh	but it'll come
[12:24] TheEffigy	i was just under the impression ephemeral ports in general were just the range of ports "reserved" but not in the commonly used range
[12:25] pieterh	http://www.tcpipguide.com/free/t_TCPIPClientEphemeralPortsandClientServerApplicatio-2.htm
[12:26] pieterh	"Just as well-known and registered port numbers are used for server processes, ephemeral port numbers are for client processes only."
[12:27] guido_g	great link, thx
[12:27] TheEffigy	ahh, yes that does make more sense now that i think about it
[12:31] pieterh	TheEffigy, another technique you can use, depending on the scenario, is dynamic port numbers
[12:32] pieterh	e.g. server can find an available port by scanning (trying to bind from 4096 up until it works)
[12:32] pieterh	and then send the endpoint info to the client via another socket
[12:33] pieterh	but in general if you're using PAIR sockets, over TCP, 90% likely you're doing your architecture wrong
[12:35] TheEffigy	yes perhaps
[12:35] TheEffigy	i'll explain
[12:36] TheEffigy	my understanding is that the pub/sub matching is based on the client side only
[12:37] TheEffigy	i'm taking a brokered approach to the system that i'm building, which is bringing in data from many sources and streaming it to connected applications
[12:37] TheEffigy	however they might not be interested in all the data, which can be easily categorised on a few levels
[12:38] TheEffigy	i'd rather not send the data onwards if they're not interested as it would only be discarded anyway, and it is potentially quite a lot of data
[12:39] pieterh	Ok, kind of a classic scenario
[12:39] pieterh	so there are several options here
[12:40] pieterh	a. wait until 0MQ does publisher-side filtering
[12:40] pieterh	b. split data up over a set of static sockets
[12:40] pieterh	c. create and manage one socket per subscriber
[12:40] TheEffigy	c. is the approach i was going for
[12:41] pieterh	first question would be, how many subscribers?
[12:41] TheEffigy	no more than 50
[12:41] pieterh	aight, so solution c would work
[12:41] pieterh	i would do it as follows
[12:42] pieterh	- req/rep socket for subscription requests
[12:42] TheEffigy	yep - done that part already
[12:42] pieterh	- one pub socket per subscriber
[12:43] pieterh	- some kind of heartbeating to allow server to clean up
[12:43] pieterh	- dynamic port selection using technique I already outlined
[12:43] pieterh	- i.e. server looks for available port within some range, then tells the client this endpoint
[12:43] TheEffigy	yep
[12:43] TheEffigy	sounds good.
[12:43] pieterh	that will work
[12:44] TheEffigy	only question i have is basically what is the difference between using a pub socket instead of a pair?
[12:44] pieterh	pair does not handle, for example, reconnections
[12:44] TheEffigy	ah
[12:44] TheEffigy	yeah, well that's enough to make it a deal breaker
[12:44] pieterh	actually rather than pub, consider using PUSH/PULL or XREP/XREQ
[12:45] pieterh	advantage of PUB is that you can connect more than one client to the server endpoint, transparently
[12:45] pieterh	if you use PUSH or XREP, then if a second client connects to the same endpoint, messages will get load-balanced
[12:45] pieterh	which is not what you want
[12:46] TheEffigy	ah
[12:46] TheEffigy	interesting.
[12:46] TheEffigy	i'll have a read up on those
[12:46] pieterh	so with PUB sockets you can do things like make subscriptions + endpoints available downstream
[12:46] pieterh	to other subscribers
[12:46] pieterh	without any risk that it'll affect existing ones
[12:46] TheEffigy	excellent
[12:46] pieterh	np
[12:47] TheEffigy	thanks so much for pointing me in the right direction
[12:47] pieterh	did you read the guide yet?
[12:48] pieterh	i know this is not covered
[12:55] TheEffigy	i did, but that was just before my holiday so i've forgotten some of the finer points. might be time for a quick refresher
[16:18] ptrb	is zmq_socket threadsafe on the context?
[16:23] pieterh	ptrb, what do you mean...?
[16:23] pieterh	in 0MQ/2.0.x you cannot use a socket except in the thread you created it in
[16:24] pieterh	this is explained in the Guide
[16:24] pieterh	in 0MQ/2.1.x you can create a socket in one thread and use it in another, but not share it between two threads at once
[16:27] ptrb	right, that's clear. I mean if I create a shared context, can multiple threads call zmq_socket(ctx, ...); on it without protection
[16:35] pieterh	ptrb: yes, certainly
[16:35] pieterh	you'll see a lot of that in the examples in the guide
[16:48] ptrb	cool, great.