ZeroMq IRC Log

Thursday June 16, 2011

[Time] Name	Message
[09:20] PaowZ	hi there ! just built the 0MQ lib for windows, but I got a dll.. is there a way to obtain a static lib, instead ? I saw libzmq.dll.a but its size seems a bit light to be the good one..
[09:39] PaowZ	..and why the dll is so big, also..
[09:47] PaowZ	when I add --enable-static to 'configure' I get this error:
[09:47] PaowZ	configure: error: building static libraries is not supported under MinGW32
[09:48] PaowZ	l.17096 in 'configure' fome
[09:48] PaowZ	file
[09:48] sustrik	i think building static libs is not supported under windows
[09:49] sustrik	you'll have to tweak the build process by hand
[09:49] PaowZ	why so ?
[09:49] sustrik	nodoby done it :)
[09:49] sustrik	you can be the one
[09:49] PaowZ	okay.. I see..
[09:49] PaowZ	googling a bit gave me some hints..
[09:49] PaowZ	a guy did it under MSVC
[09:50] PaowZ	but I only run MinGW environment..
[09:50] sustrik	ok
[09:51] PaowZ	another question, why the dll is so fat ?
[09:51] PaowZ	2Mb.. it's pretty heavy..
[09:52] PaowZ	this link is interesting anyway.. https://github.com/zeromq/erlzmq/commit/ab65e4897fcb9b03cbea7df3dbc7ca872018175b
[09:53] PaowZ	some line adds in the configure.ac file.;
[09:53] PaowZ	however, I can't file such a file in the current archive of zeromq
[09:55] klogd	anyone gotten the java version of the example of the durable subscriber/publisher (from ÃMQ - the guide)?
[09:55] klogd	I run them, but "durasub" does not continue where it stopped, so several messages are lost
[09:56] mikko	klogd: they might get lost in network buffers
[09:56] mikko	but sustrik_ can probably clarify it more
[10:00] klogd	seems there is an error in the java example.. subscriber.setIdentity() is never called
[10:00] klogd	should I report it somewhere?
[10:01] sustrik	yep, send the patch to the mailing list
[10:01] sustrik	or send a pull request to zguide project
[10:01] PaowZ	sustrik_, what's your opinion about this ? https://github.com/zeromq/erlzmq/commit/ab65e4897fcb9b03cbea7df3dbc7ca872018175b
[10:02] PaowZ	what's configure.ac file ?
[10:02] sustrik	no idea
[10:02] sustrik	isn't that a generated file?
[10:02] sustrik	mikko: any idea?
[10:03] PaowZ	sustrik_, I don't guess at first sight.. I thought about a configure file, just like the ones you can find in the archive (i.e.: configure.in and configure)
[10:25] mikko	checking
[10:26] mikko	configure.ac is not autogenerated
[10:26] mikko	the file is called .in or .ac usually
[10:27] PaowZ	I tried to configure the thing using this .ac
[10:27] PaowZ	but the git returns me an error, can't get the sources..
[10:28] PaowZ	I think I'm close to tweak the build myself to build the static lib..
[10:29] Seta00	I thought the MinGW build generated .la files/
[10:30] PaowZ	indeed, I saw such a file..
[10:30] PaowZ	but it's not a static lib.. is it ?
[10:30] mikko	.la is libtool archive
[10:31] PaowZ	hm..ok
[10:31] PaowZ	..be back..
[10:31] Seta00	.la archives have both static and dynamic libs
[10:32] PaowZ	you mean you can open it up and extract the lib you need ?
[10:32] mikko	Seta00: depends on the platform
[10:32] Seta00	mikko, yeah, I guess you can use it just as a .a archive
[10:32] PaowZ	under windows, I actuaaly am..
[10:32] mikko	you cant
[10:32] Seta00	PaowZ, there's a proper way to link to it
[10:32] mikko	take a look at .la file contents
[10:33] PaowZ	..gotta go.. I'll be right back..
[10:33] mikko	please stop changing your nick
[10:33] Seta00	yeah, just add the .la file to LDADD
[10:33] Seta00	that should do the trick
[10:34] mikko	yes, but that doesnt help if static libs are not built
[11:55] pieterh	sustrik: ping?
[11:55] sustrik	pong
[11:55] sustrik	hi
[11:56] pieterh	hey, how's it going?
[11:56] sustrik	fine
[11:56] sustrik	beginning to be a bit hot here
[11:56] pieterh	I'm reading Helge Frueh's question on email
[11:56] sustrik	the IP addresses of the peers?
[11:57] pieterh	I was wondering why we don't use remote hostaddr:port as a natural key
[11:57] pieterh	rather than generating a UUID
[11:57] sustrik	how would that work with non-TCP transports?
[11:57] pieterh	it'd be transport specific, obviously
[11:58] pieterh	but all IP-based transports use hostname:port afaik
[11:58] sustrik	uuid is nice as it's universal
[11:58] pieterh	or some combination thereof (could be IPv6)
[11:58] pieterh	well, UUID hides this information, I was wondering if there was a scalability argument for that
[11:58] sustrik	not really
[11:59] pieterh	nice
[11:59] sustrik	the only requirement on the field is to be unique
[11:59] pieterh	yes... for all IP-based transports, this will be unique and transient
[11:59] pieterh	i.e. each connection creates a new port at the other peer
[11:59] pieterh	ports may be recycled, that's the possible downside
[12:00] sustrik	the only question is what it would be good for
[12:00] pieterh	well, I see a couple of benefits
[12:00] sustrik	users should treat the field as an opaque unique identifier anyway
[12:00] pieterh	one, you get printable and sane addressing
[12:00] pieterh	which can be extremely useful for e.g. tracing what a message is doing
[12:01] pieterh	since the backtrace will show the actual ip addresses of each node
[12:01] pieterh	two, you allow IP-based routing in devices
[12:01] sustrik	?
[12:01] pieterh	which is extremely valuable for some use cases
[12:01] pieterh	e.g. "this address range is banned"
[12:01] sustrik	it's an opaque identifier
[12:02] pieterh	well, that was my question
[12:02] pieterh	is there a specific scalability reason for making it opaque?
[12:02] so_solid_moo	what would happen to people with rfc1918 addresses behind natting firewalls? Those aren't unique IPs.
[12:02] sustrik	you bound the whole topology to particular underlying transport
[12:02] sustrik	like tcp
[12:03] sustrik	so, if you use a differnet transport at any hop
[12:03] sustrik	(say native IB or somesuch)
[12:03] sustrik	you confuse the devices on the path
[12:03] so_solid_moo	pieterh: I'm talking about people with 192.168.0.* subnets
[12:03] pieterh	sustrik: hang on, let me understand so_solid_moo's question
[12:04] pieterh	so_solid_moo: they'll come in as distinct ports on the peer
[12:04] pieterh	afaik
[12:04] so_solid_moo	say, if you have two offices hanging off adsl, and you have zeromq clients hooking up to some internet server
[12:04] sustrik	pieterh means source ports afaiu
[12:04] so_solid_moo	and both offices have the same Zytek or whatever adsl modem
[12:04] pieterh	when you get a TCP connection you'll have a host:port at the other end
[12:04] pieterh	ditto when you get a UDP message
[12:05] so_solid_moo	so it's the host:port of the firewall?
[12:05] pieterh	sure
[12:05] pieterh	that's fine, it's unique and allows routing back
[12:05] pieterh	so, sustrik, back to your argument about non-conformity
[12:05] pieterh	I think this could simply be handled as for transport names
[12:06] pieterh	i.e. tcp://hostname:port, udp://hostname:port, ib://whatever
[12:06] pieterh	applications can parse what they need to
[12:06] pieterh	more valuably, they can log stuff in a useful way
[12:06] sustrik	well, i would rather keep it opaque
[12:06] pieterh	inproc://whateber
[12:06] sustrik	for example
[12:06] sustrik	i would like to use 16bit ids in the buture
[12:07] so_solid_moo	ip:port is unique, hostname:port isn't necessarily
[12:07] sustrik	as an bandwidth optimisation
[12:07] pieterh	sure, that's an obvious optimization
[12:07] sustrik	once you hard-wire the format
[12:07] sustrik	you can't do that kind of thing
[12:07] pieterh	sustrik: no, you don't hardwire it, you allow the transport to use whatever format it wants
[12:07] pieterh	so_solid_moo: it's the full string that's unique
[12:07] sustrik	sure, that's ok
[12:08] pieterh	ok, let me make a short proposal to the list
[12:08] pieterh	I think this could be fun and useful
[12:08] sustrik	the problem is that one node cannot reply on other node to use the same format
[12:08] so_solid_moo	pieterh: if it's a dns hostname it won't necessarily be unique, though, was my point
[12:08] sustrik	so it's ok far tracking
[12:08] pieterh	so_solid_moo: the dns name plus port is what has to be unique
[12:08] sustrik	not ok for business logic or routing
[12:08] so_solid_moo	pieterh: but that wouldn't be
[12:08] so_solid_moo	I know for sure that isn't on my network.
[12:09] pieterh	so_solid_moo: you mean you'll get multiple incoming TCP connections that appear to speak to the same remote port?
[12:09] so_solid_moo	pieterh: no, because connections work on IP, not hostname
[12:09] so_solid_moo	translating the IP to a hostname isn't a reversible operation
[12:09] pieterh	sure
[12:09] pieterh	it's the IP:port that you get
[12:09] pieterh	forget dns, it's not relevant here
[12:10] so_solid_moo	ok, it's just you were mentioning 'hostname' not 'IP'
[12:10] pieterh	I should have written hostaddr:port, sorrty
[12:10] pieterh	sorry
[12:10] so_solid_moo	ah, ok
[12:11] so_solid_moo	maybe not ':' too, that could be ambiguous for ipv6 I think ;)
[12:11] so_solid_moo	(it wouldn't be, but it would look ambiguous)
[12:18] pieterh	so_solid_moo: we'd probably want an ipv6 scheme, no?
[12:18] pieterh	sustrik: I have a proposal for a new ROUTER API
[12:19] sustrik	ok
[12:19] pieterh	Sent to the list but basically it's orthogonal with getting the sender address
[12:20] pieterh	it would neatly split ROUTER away from XREP
[12:20] so_solid_moo	pieterh: yeah, definitely, just that something like fe80::213:23 looks like an IPv6 address
[12:20] so_solid_moo	when you'd really mean fe80::213 port 23
[12:20] pieterh	hehe, you could probably continue to use opaque identities for request-reply routing
[12:20] pieterh	so_solid_moo: I'
[12:20] pieterh	I'll be playing with IPv6 in a short while
[12:23] pieterh	sustrik: I think this is the way to split ROUTER from XREP
[12:23] sustrik	the proposal is about getting identities
[12:23] pieterh	precisely
[12:23] sustrik	how does that realte to socket types?
[12:24] pieterh	?
[12:24] pieterh	xrep needs a stack
[12:24] pieterh	router doesn't, that's a clumsy api
[12:24] pieterh	router needs to get an identity
[12:24] pieterh	and then also set an identity when sending
[12:24] pieterh	if you already provide a get identity function for all receiving socket types
[12:25] sustrik	ok, so we'll split the ROUTER pattern from REQ/REP
[12:25] pieterh	then you have half the API for a new router socket type
[12:25] sustrik	right?
[12:25] pieterh	yes and no
[12:25] pieterh	yes and no
[12:25] pieterh	we split router from the req/rep stack business
[12:25] pieterh	completely orthogonal
[12:25] pieterh	however, router only makes sense as part of the overall request-reply pattern package
[12:25] sustrik	that won't work
[12:26] pieterh	i can't connect router to router, or pull, or push, or pub, or sub, or pair
[12:26] sustrik	the routing functionality has to be encapsulated in req/rep pattern
[12:26] sustrik	as the routing is the service the pattern provides to the user
[12:26] pieterh	perhaps you didn't understand what I'm proposing
[12:26] sustrik	probably not
[12:26] pieterh	shall we try again or do you want to do this later?
[12:27] pieterh	i am very patient :)
[12:27] sustrik	send the proposal to the ML
[12:27] pieterh	already did that
[12:27] sustrik	there's nothing about XREP/ROUTER problem there
[12:29] pieterh	perhaps I'm not restating the problem as you believe it exists, but that's not my job
[12:29] sustrik	ah, you mean the last paragraph
[12:29] pieterh	and the first proposal
[12:30] sustrik	well, the problem remains, the proposal just adds a little syntactic suger on top
[12:30] pieterh	what is the 'problem'?
[12:30] sustrik	the problem is that the service the request/reply pattern delivers
[12:30] pieterh	and this is not syntactic sugar, please don't do this when you don't understand a proposal
[12:30] pieterh	'it's a hack'
[12:31] sustrik	is opaque routing of requests to workers and replies back to the requesters
[12:31] pieterh	sigh
[12:31] sustrik	user is not supposed to care about it
[12:31] klogd	Where does ZMQ keep the swap file? is there a way to change the location/drive ?
[12:31] pieterh	it is so difficult to discuss this when you divorce yourself from real use cases so efficiently
[12:31] pieterh	klogd: nope
[12:32] sustrik	i had a look at lruqueue
[12:32] pieterh	you also need to ask users what they actually do
[12:32] klogd	pieterh: didn't think so, thanks .)
[12:32] sustrik	what it does is introducing a new scheduling mechanism
[12:32] pieterh	not scheduling
[12:32] pieterh	opaque routing
[12:32] pieterh	scheduling?
[12:32] sustrik	like "who's going to process next request"
[12:32] pieterh	that's not scheduling
[12:32] pieterh	scheduling is "when X will happen"
[12:33] klogd	can the filename of the swap file be predicted?
[12:33] sustrik	ok, call it something else
[12:33] pieterh	this is just another type of routing, beside lb
[12:33] pieterh	routing
[12:33] pieterh	it's a routing device
[12:33] sustrik	any kind of routing is ok
[12:33] pieterh	we've been discussing this for years now
[12:33] sustrik	the point is that it's transparent
[12:33] pieterh	devices sometimes NEED to be able to address specific connections
[12:34] sustrik	ack
[12:34] pieterh	well, you never provided a way for this to work
[12:34] sustrik	but the end user should not
[12:34] pieterh	so we abused XREP
[12:34] pieterh	end-users never use xrep
[12:34] pieterh	that's a ghastly assumption to make
[12:34] pieterh	it assumes there's no layering in code that builds on 0mq
[12:34] pieterh	whereas every single project always layers itself, it has to
[12:35] pieterh	there is no use case for routing in application code, that should be clear
[12:35] pieterh	any kind of routing
[12:35] sustrik	right
[12:35] pieterh	with the exception of in-process routers
[12:35] pieterh	for peer-to-peer, which we also do (freelance pattern)
[12:35] pieterh	still cleanly separated from any application code
[12:36] sustrik	ok, so my idea of how custom routing should work is
[12:36] sustrik	for example:
[12:36] sustrik	add a socket option to select a "scheduler" type
[12:36] sustrik	(call it whatever you want)
[12:36] sustrik	or
[12:36] sustrik	special implementation of the device
[12:36] sustrik	say a FPGA device
[12:37] pieterh	it's "router", a scheduler is a program that arranges jobs or a computer's operations into an appropriate sequence
[12:37] sustrik	ok, good
[12:37] sustrik	then you can have a "ROUTER" socket option to choose an routing strategy
[12:37] pieterh	so the problem with your idea, which is fine, is that it doesn't actually solve routing as used in 0MQ applications
[12:37] pieterh	it's not about strategy
[12:38] pieterh	it's about talking to connections
[12:38] pieterh	shall I go through the various use cases?
[12:38] sustrik	no need
[12:38] pieterh	e.g. authentication
[12:38] sustrik	my point is that you should never speak to particulat connection
[12:38] pieterh	heartbeating and cleaning up dead peers
[12:38] pieterh	my point is that you should
[12:39] sustrik	0mq layer should abstract you from that
[12:39] pieterh	you can't reject a significant proportion of 0MQ use cases with "should not"
[12:39] sustrik	that's the service it provides
[12:39] pieterh	it cannot until it becomes bigger than AMQP
[12:39] sustrik	"you don't have to care about connecitons"
[12:39] pieterh	which you don't IMO want to become
[12:39] pieterh	"but when you have to care, use ROUTER"
[12:39] pieterh	that's the whole point, Martin
[12:39] pieterh	you cannot delete the question
[12:40] pieterh	you will simply annoy and alienate your users
[12:40] sustrik	so what i proposed is to create a new pattern
[12:40] sustrik	that allows for explicit trouting
[12:40] sustrik	that way people wouldn't be able to break req/rep functionality
[12:40] sustrik	by doing insane things
[12:40] pieterh	you're not doing a good job of explaining why the current approach is broken
[12:40] pieterh	calling it 'insane' is not scientific
[12:40] sustrik	take an example of TCP
[12:40] pieterh	nor is there any breakage
[12:40] pieterh	no, no examples
[12:41] pieterh	what is actually broken today?
[12:41] pieterh	your opinion is not science
[12:41] sustrik	imagine i have an requirement to send any packet to the peer
[12:41] pieterh	nor is mine, obviously
[12:41] sustrik	if i add that to TCP stacks
[12:41] pieterh	imagine?
[12:41] sustrik	the user would be able to break the TCP functionality
[12:41] sustrik	by inserting random packets to the flow
[12:41] sustrik	it breaks the encapsulation of the functionalirt
[12:42] sustrik	same thing with req/rep
[12:42] pieterh	this is not concrete, it's a theoretical argument against pragmatic use
[12:42] pieterh	won't sell
[12:42] pieterh	doesn't sell
[12:42] sustrik	i can guarantee that the message is delivered to a worker and the reply gets back to the requester
[12:42] pieterh	ok, let's pause this
[12:42] sustrik	but to do that
[12:42] pieterh	we're stuck in philosophy and I want to make code
[12:42] sustrik	i have to restrict users in some ways
[12:43] pieterh	you can't guarantee anything
[12:43] pieterh	have you ever tried to make a reliable application?
[12:43] pieterh	its... kind of difficult
[12:43] pieterh	especially with 0MQ
[12:43] pieterh	there is in fact an utter and total lack of guarantees
[12:43] pieterh	charming, kind of
[12:44] pieterh	you don't guarantee any deliveries
[12:44] sustrik	not yet
[12:44] sustrik	to be done
[12:44] pieterh	in fact using ROUTER is largely to make that work properly
[12:44] pieterh	sigh
[12:45] pieterh	I can guarantee you that significant parts of 0MQ are over-engineered
[12:45] pieterh	while other vital parts are under-developed
[12:45] sustrik	basides the point
[12:45] pieterh	no, fully relevant
[12:45] sustrik	the problem we are talking about is layering
[12:45] pieterh	your biases are not helpful
[12:45] pieterh	your view of layers is inaccurate
[12:45] pieterh	it's based on theory, not practice
[12:45] sustrik	my point of view is that routing should be done inside 0mq
[12:45] pieterh	I can guarantee you it's inaccurate, and you'll see that one day
[12:45] sustrik	you want to have it on top
[12:46] sustrik	that's what's the whole discussion is about
[12:46] pieterh	in the meantime, you'll make stuff that's really good at some things, and horribly flawed elsewhere
[12:46] pieterh	if you actually used the tool, to make software, you'd see that faster
[12:46] pieterh	listening carefully to your users also helps
[12:46] pieterh	trying to educate them doesn't
[12:46] sustrik	sure, there's functionality missing
[12:47] pieterh	that's not what I'm saying, Martin
[12:47] pieterh	what you have made is in significant parts still inaccurate
[12:47] sustrik	but trying to fix it by adding layers on top doesn't make sense imo
[12:47] pieterh	and you defend the inaccuracies vehemently, as all designers do with their work
[12:48] sustrik	ok, no point in this discussion
[12:48] pieterh	thanks
[12:49] pieterh	anyone? someone? surely someone uses this?
[12:49] pieterh	but if almost no-one does multi-hop request-reply, why do we have all that complexity...?
[14:47] PaowZ	ok.. I got the lib to compile statically under windows.. roughly weighted, by the way..
[14:54] sustrik	PaowZ: try sending the patch to the mailing list
[14:55] PaowZ	sustrik, actually it's more of a question of switch to supply to the compiler than real patching..
[14:56] sustrik	at least describe how you did it
[14:56] sustrik	others may incorporate the switch to the build system later on
[14:57] ssi	pieterh: re multi-hop: I'm doing an LRU queue now, does that count as multi-hop?
[14:57] PaowZ	I just got platform.hpp copied in the needed directory and told the compiler the switch ZMQ_HAVE_MINGW.; I must not be the first one to do such a thing.. Else, I shall test the effiency of the obtained library before sharing, I guess..
[14:58] pieterh	ssi: we're using 'multi-hop request reply' to mean going across a zmq_queue device, strictly
[14:59] pieterh	i.e. routing replies only using the address stack
[15:00] ssi	gotcha
[15:02] ssi	so, tell me about: Assertion failed: ok (xrep.cpp:62)
[15:02] ssi	I seem to get that intermittently
[15:02] ssi	I have a sneaking suspicion it has to do with closing and reopening sockets
[15:06] pieterh	you're using explicit identities, right?
[15:11] pieterh	ssi: see ^^^
[15:13] sustrik	ssi: what version is that?
[15:15] truemped	Hi everyone! I have a very quick question regarding pyzmq. Somehow I am unable to build it...
[15:16] truemped	It always stops with the error: "pyzmq/zmq/core/message.pyx:290:30: Cannot convert 1 argument from 'int ' to 'Py_ssize_t '"
[15:17] truemped	Any ideas?
[15:31] truemped	Answering myself: it was an old cython version...
[16:47] ssi	sorry for the delay, got meeting'd. I was using explicit identities, but I took them out and have the same issue
[16:47] ssi	and I'm on 2.1.7
[16:55] sustrik	ssi: ok, do you have a reproducible test case?
[17:09] ssi	sustrik: not at the moment
[18:08] pieterh	ssi: re
[18:08] pieterh	this assert is caused by more than one client connecting with the same identity
[18:08] pieterh	that can only happen if you're (mis)using explicit identities
[18:09] pieterh	ergo, you're still running code that's using explicit identities
[18:09] pieterh	if you want to double-check, just add a printf to libzmq before it asserts
[18:09] pieterh	this is one of those useless asserts that tells you absolutely nothing... :-/
[18:13] ssi	peer_identity_, is that a string that I can print directly?
[18:14] ssi	doesn't look like it...
[18:16] ssi	ok so lemme ask you this... can I create a socket, set its identity, and then connect/close it as needed?
[18:16] ssi	or does close destroy it
[18:17] ssi	I'm running into open file limits pretty quickly, and I want to be able to keep only as many sockets open as absolutely necessary
[18:29] john	Does anyone here use Protocol Buffers in Python? I've been reading a bit about it and in the README it claims to be fairly slow and buggy in the Python implementation. Have any of you experienced bugs or slowness?
[18:30] pieterh	ssi: you can raise the file limits, depends on the OS
[18:30] pieterh	you should usually not be opening/closing sockets
[18:31] pieterh	once you close a socket, you destroy it
[18:31] pieterh	*not be opening and closing sockets all the time
[18:32] ssi	ok
[18:53] ssi	out of curiosity, why do inproc sockets use file handles?
[19:06] pieterh	ssi: that's a good question, kind of strange, isn't it...
[19:06] pieterh	IMO it was a change introduced to get over a limit in number of inproc connections
[19:10] sustrik	every socket uses a "mailbox" to communicate between application thread on i/o threads
[19:10] sustrik	the mailbox is a socketpair
[19:10] sustrik	thus it consists of 2 fds
[19:10] ssi	for inproc, are they stream sockets?
[19:11] sustrik	it's a socketpair
[19:11] sustrik	man socketpair
[19:12] sustrik	it is not dependent on what underlying transports there are underneath the socket
[19:12] pieterh	sustrik: just wondering, in some tests I get the same performance for inproc as for TCP
[19:12] pieterh	something like 6M events per second
[19:13] pieterh	is that what you'd expect?
[19:13] sustrik	i've never tested IPC
[19:13] pieterh	inproc, not IPC...
[19:13] sustrik	ah
[19:14] sustrik	small messages, right?
[19:14] pieterh	presumably, yes
[19:14] sustrik	for small messages the network bandwidthe is not the limiting factor
[19:15] pieterh	understood, this is number of events per second, not bandwidth
[19:15] sustrik	it's per-message overhead instead
[19:15] pieterh	I thought inproc didn't go through the kernel
[19:15] sustrik	what's event?
[19:15] pieterh	a recv or a send
[19:15] sustrik	it does not
[19:15] pieterh	i.e. pure reader can do 6M, device can do 3M per second
[19:16] sustrik	the bottlenect in case of small messages is synchronisation
[19:16] sustrik	not the actual transport mechanism
[19:16] pieterh	there seems to be some batching effect, but not massive
[19:16] sustrik	so it's likely to be the same for all transports
[19:17] pieterh	where's the synchronization bottleneck?
[19:17] sustrik	atomic ops to pass messages between threads
[19:17] sustrik	or
[19:17] sustrik	amount of work done per message
[19:17] pieterh	right
[19:17] sustrik	which is ~150ns
[19:18] sustrik	given that an CPU tick is 0.3 ns
[19:18] sustrik	it's pretty good
[19:18] pieterh	so the advantage of inproc over tcp is less copying, I guess?
[19:18] sustrik	yes
[19:18] sustrik	but that's irrelevant for small messages
[19:19] pieterh	does it make sense then to batch at the application level, if using inproc?
[19:19] sustrik	unlikely you can do it better than 0mq
[19:19] sustrik	you are free to try though
[19:20] pieterh	indeed, but these are questions people ask
[19:20] sustrik	in any case getting per message overhead below 150 ns in not easy
[19:20] sustrik	every single CPU instruction takes couple of ns
[19:20] pieterh	indeed
[19:20] ssi	is there an upper limit on single msg size?
[19:21] ssi	imposed or practical?
[19:21] sustrik	2^64
[19:21] pieterh	ssi: available memory
[19:21] pieterh	:)
[19:21] ssi	ok
[19:58] ssi	hrm
[19:58] ssi	the way I'm constructing my socket identities is by using my thread name (which is unique per worker)
[19:59] ssi	so my identity might be something like W-showMessage-worker-1234
[19:59] ssi	is it possible that there's an identity size limit and it's getting clipped, causing duplicates?
[20:00] ssi	man page says 255 bytes, so I guess not
[20:00] sustrik	why do you do it at all?
[20:01] sustrik	0mq generates an identity automatically for you if you don't set one
[20:01] ssi	I originally set the identities to make debugging the pattern easier
[20:01] ssi	and it seems like I get the assertion failure if I take the identities OUT
[20:01] ssi	because I think I'd rather NOT have identities... I don't want it to do anything crazy across restarts
[20:02] ssi	hrm no, got the assertion failure there and I have identities
[20:02] ssi	it's very intermittent :/
[20:03] sustrik	retport the bug then
[20:03] sustrik	report
[20:04] ssi	I'd need a test case to do that, and it's likely that I'm just doing something wrong
[20:05] sustrik	you shouldn't be able to cause assert
[20:05] ssi	I probably just need to spend the time studying the zmq so I can figure out how to print the identity of the socket that's failing the assertion check, so that I can debug what my code is doing stupidly
[20:05] sustrik	if you see an assert its 0mq bug
[20:05] sustrik	unless you use sockets from multiple threads that is
[20:06] ssi	that's possible! I have gone out of my way not to, but I may be screwing it up
[20:07] ssi	ah there's one other place where I'm closing and reopening a socket... lemme change that back
[20:10] ssi	that may have done it
[20:45] pieterh	sustrik: you recall that patch I sent that logged the identity for this error, instead of asserting?
[20:46] pieterh	ssi: here's code that prints the identity (it's from an old patch that got slashed)
[20:46] pieterh	https://gist.github.com/1030232
[22:30] ssi	pieterh: awesome, thanks
[22:50] jhawk28	pieterh: I gave my first 10 minute ZMQ talk yesterday. It turned into an hour discussion afterwards