ZeroMq IRC Log

Friday September 17, 2010

[Time] Name	Message
[01:41] mattp	Hi. I'm trying to build using Cygwin. I'm getting me an error message telling me to install e2fsprog even though I've already got it.
[01:41] mattp	Any suggestions?
[01:42] mattp	(not sure how that "me" got into my third sentence. :) )
[01:49] mattp	crickets...
[01:52] mpassell	thought I'd try through a different client. Anyone out there?
[03:51] chowmeined	can one multiplex different message patterns on a single socket?
[04:18] andrewvc	cremes, by chance are you around?
[04:33] dermoth	iirc there used to be device config example, looks like they're gone. why?
[04:33] dermoth	king of easier than reading the code :)
[04:38] guido_g	dermoth: see http://github.com/imatix/zdevices
[04:38] guido_g	maybe thats what you want
[04:40] dermoth	guido_g, thanks, looks nice. Actually for now i'm just using the stock devices that come with zeromq. The config quite basic, in xml. I was justt wondering where they gone
[04:41] guido_g	afair there going to be moved out of the core distribution
[04:42] dermoth	ok... well I kind of like them for the most basic usage... I could replace the config with command-line arguments though, imho it would be much easier
[04:43] guido_g	dermoth: btw, please send your patch to the ml with a statement that it is licensed under mit/x11, othwerwise it can't be accepted
[04:43] dermoth	really?
[04:44] guido_g	it's the procedure
[04:44] guido_g	has been discussed here and on the ml
[04:46] guido_g	dermoth: easiest way is to use format-patch and append that to the mal w/ the license statement
[04:46] dermoth	then how about I sen you this?
[04:46] dermoth	perl -pi -e 's/ZMQ_DOWNSTREAM/ZMQ_PUSH/; s/ZMQ_UPSTREAM/ZMQ_PULL/;' devices/zmq_streamer/zmq_streamer.cpp
[04:46] guido_g	not me
[04:46] guido_g	dermoth: see git format-patch
[04:46] dermoth	git commit -m'zmq_streamer: Rename DOWNSTREAM/UPSTREAM to PUSH/PULL'
[04:47] dermoth	I use git-format-patch... but I'm not even subscrived to the ml
[04:47] guido_g	too bad
[04:47] dermoth	the two commands abot will get you the same result less the credit
[04:48] guido_g	what credit?
[04:48] dermoth	the author in commit message
[04:48] guido_g	ah
[04:48] dermoth	unless you're authorised to do: git commit -m'zmq_streamer: Rename DOWNSTREAM/UPSTREAM to PUSH/PULL' '--author=Thomas Guyot-Sionnest <thomas@blinkx.com>'
[04:48] guido_g	i'm more concerned about the license statement, w/o it the patch can't go in
[04:49] dermoth	it's not a patch it a command that patsh, there shouldn't be any legal issue. if you run the command you have the copyright for the resulting code ;)
[04:50] guido_g	i think that is clear
[04:50] guido_g	i just stated what's needed to get a patch in, because a pull-req is a patch
[04:54] dermoth	I understand the legal basis, but I will likely not send a patch that takes me more time to "send" thna write... Imagine trying that with the Linux Kernel!
[04:54] guido_g	as i said, the device apps will be moved out of the core, afaik
[04:55] guido_g	so...
[04:55] dermoth	so I guess there won't be any trouble getting that in, whatever method you use... on that i'll get back to my testing :)
[04:55] guido_g	dermoth: wrong
[04:55] guido_g	the license issue is a problem atm
[04:55] guido_g	ask pieter about it
[04:56] guido_g	and go read the ml (via gmane w/o registration)
[04:56] dermoth	what I meant is that it probably take you less time to rename it yourself than using my patch (and I use gmane...)
[04:57] dermoth	i wouldn't mind taking the time to post on the ml for a real change... and can'T I put it in the commit message too?
[04:58] guido_g	cool, then you can easily find the threads about patches and legal issues
[04:58] guido_g	if it were in the commit message, even better
[04:58] dermoth	except that i'd rather catch some much needed sleep
[04:58] dermoth	ok
[05:04] dermoth	here you go... next time i'll know :)
[05:12] guido_g	hey, i didn't invent the laws and such, just repeating the handbook
[05:13] dermoth	Any PECL developers here? IMHO the PHP api should have a cb that is called on persistent socket creation, so we could create the socket and connect in one shot
[05:39] sleeperbot	anybody know an alternative driver to http://github.com/JustinTulloss/zeromq.node for node.js?
[05:40] sleeperbot	this driver takes up a ton of CPU
[05:42] sleeperbot	the way that it is implemented =/
[05:48] yrashk	sleeperbot afair there's none
[05:48] yrashk	but supposedly I will need one soon-ish
[05:48] yrashk	so might invest some time in getting it done properly
[05:51] sleeperbot	i see, thanks for the info
[05:52] yrashk	yw
[07:02] sustrik	psino: how would it work for recv calls?
[07:03] sustrik	dermoth: it's a problem with OpenPGM, which will be solved soon; what i was saying is that in most cases you have just one context so you never encounter the problem
[07:04] sustrik	chowmeined: no, you can't
[08:49] ivan-korobkov	Hi, I use zmq 2.0.9 + epgm + the latest python bindings. Sometimes it crashes with Assertion failed: *tmpbuf > 0 (zmq_decoder.cpp:60). Does anybody know why it happens? Is it a bug and I should create an issue?
[09:08] guido_g	try to isolate it in a small test case, so others can verify it
[09:12] ivan-korobkov	OK, I will try, but it happens rather sporadically.
[09:13] guido_g	will be no problem, if we have a program that can run
[09:31] psino	sustrik: In the case of a pull socket, I was thinking it could disconnect from the push socket and still be able to recv() on the messages the push socket has sent to it
[09:40] sustrik	ivan-korobkov: create an issue
[09:41] sustrik	psino: any idea of how the API should look like?
[09:46] psino	sustrik: from the end users point of view, much like now, except that after calling socket.close(), a poller wouldn't put the socket in exceptional state before the messages that has been sent to it from the other side has been recv()'ed
[10:32] CIA-20	zeromq2: 03Martin Sustrik 07master * r85aa25e 10/ src/req.cpp : bug in REQ+multipart fixed - http://bit.ly/bzWD3V
[10:58] CIA-20	zeromq2: 03Martin Sustrik 07master * rc7b9ba3 10/ (src/command.hpp src/session.cpp): reconnection process fixed when failure occurs during init phase - http://bit.ly/cIBUcl
[20:02] bgranger	sustrik: question about msg ref counting...
[20:11] keffo	I think they're all off work :)
[20:11] keffo	starting to think I'm the only one around anymore :)
[20:12] keffo	lol, last time anyone even spoke in here was about 11h ago :)
[20:12] andrewvc	and you just broke the record
[20:12] keffo	no, bgranger did 10 mins ago :)
[20:14] andrewvc	oh damn, hehe, that's what I get for sleeping my laptop, missing all the action
[20:16] keffo	:)
[20:17] keffo	Does anyone know best-practices for transferring large amounts of data? Simply to split into multipart?
[20:17] andrewvc	I remember there were benchmarks on the zeromq site with various message sizes, ranging up to something huge
[20:18] keffo	right, but simply sending one huge message just sounds like mem hog to me
[20:32] sustrik	i am back
[20:32] sustrik	bgranger: what's the question?
[20:34] sustrik	keffo: so is multipart message, you have to break it into sequence of messages
[20:34] sustrik	then you have to define your own transfer protocol etc.
[20:34] sustrik	it's not easy
[20:34] keffo	mm
[20:35] sustrik	what do you want to do?
[20:35] keffo	To be honest I'm leaning towards an accompanying torrent lib for larger transfers
[20:35] keffo	I solved both my deadlock and garbled data btw..
[20:35] keffo	as expected it was all me :)
[20:36] keffo	CRT mismatch actually
[20:36] sustrik	:)
[20:36] sustrik	yeah, makes sense, you would have to implement torrent kind of behaiour yourself on top of 0mq
[20:36] bgranger	sustrik: we have a very subtle question about ref counting of msgs...
[20:37] keffo	sustrik, I was thinking more along the lines of using the transmission codebase, or libtorrent..
[20:37] sustrik	bgranger: yes?
[20:38] bgranger	Hi
[20:38] sustrik	keffo: sure
[20:38] sustrik	hi
[20:38] bgranger	We are trying to understand the refcounting
[20:38] bgranger	We see that zmq_msg_close is the place where the ffn is called if the ref count is 0
[20:39] bgranger	What exactly does zmq_send do with the ref count. What is zmq_close is called before a msg is actually send?
[20:39] bgranger	Does send somehow end up calling zmq_msg_close?
[20:42] sustrik	yes, send takes ownership of your message
[20:43] bgranger	OK, we just saw that at the end of xsend there is
[20:43] sustrik	your message object will be empty after return from the send call
[20:43] bgranger	int rc = zmq_msg_init (msg_);
[20:43] sustrik	yes, that initialises your message to empty
[20:43] bgranger	So if you then call zmq_msg_close you are not closing the original msg, but a new empty one.
[20:43] bgranger	Great!
[20:43] sustrik	exactly
[20:43] bgranger	We are getting python to know when all copies of a message are done being sent.
[20:44] sustrik	intercepting the free function?
[20:44] bgranger	There is a subtle issue with non-copy sends. The user has to be able to know when zmq is done with it and they can edit the buffer again.
[20:44] bgranger	Yes, we use the ffn.
[20:45] bgranger	Otherwise the user might accidentally start the edit the buffer they sent before zmq has send it...
[20:45] bgranger	non-copy sends are super subtle...
[20:45] sustrik	so you are basically writing your own allocator
[20:46] bgranger	We use Pythons garbage collection mechanism along with our own ref counts that allow us to track the underlying zmq ref counts.
[20:46] sustrik	what's the goal?
[20:47] bgranger	In Python you might have a mutable array of numbers that you want to send using non-copy.
[20:47] bgranger	You send it, and maybe then you want to modify it.
[20:47] sustrik	i see
[20:47] bgranger	But if you modify it before zmq sends it, the wrong thing gets sent
[20:47] bgranger	This is common in Python with numerical arrays (matrices, etc.)
[20:48] bgranger	For large static data it is not a problem, but in numerical/scientific computing, most data structures are mutable...
[20:49] bgranger	sustrik: I think we understand the refcounting better now though. Thanks!
[20:50] sustrik	np
[20:51] sustrik	a question though:
[20:51] sustrik	doesn't it turn the messaging into sync model?
[20:51] sustrik	one-message-a-time?
[21:08] bgranger	sustrik: back from lunch...
[21:09] bgranger	no the messaging is still fully async
[21:09] minrk	sustrik: I'm not sure what your question means. All we did was add an object that tracks whether a message no longer has references in zmq
[21:09] bgranger	You can then query that object to learn if the message has fully been sent.
[21:10] bgranger	send does not block any more than it used to
[21:10] minrk	or if there are any message objects pointing to it at all
[21:11] bgranger	minrk: it would be nice if we could have a method of MessageTracker that would block until the msg is not pending though.
[21:12] minrk	that's easy
[21:13] bgranger	minrk: I am ready when you are....just give me a call on skype
[21:13] minrk	okay
[21:14] minrk	give me a few minutes
[21:15] bgranger	no problem...
[21:20] minrk	okay
[21:22] mrossi	anyone know who is on the sending side of the signaler instance inside of the app_thread class?
[21:24] mrossi	have an app that occasionally gets hung at line 88 of app_thread.cpp doing a signaler.recv()
[21:38] dbudworth	question from a newbie zeromq user. I'd like to implement a socket server type system where the server is stateful for each client connection. Is there a zeromq socket type for this? it appears as though the server doesn't have any context about which client it's sending to at any one time
[22:51] ModusPwnens	Hmm, can anyone take a look at some results from benchmarking tests?
[22:52] ModusPwnens	I am seeing some strange results, but maybe they are normal
[22:52] ModusPwnens	http://pastie.org/1161267