ZeroMq IRC Log

Friday October 1, 2010

[Time] Name	Message
[04:20] twittard	can I use zeromq to talk to applications with a proprietary message structure
[04:20] twittard	?
[04:23] guido_g	what does that mean?
[04:34] guido_g	shrug
[04:38] Pe_Ell	I'm hoping from the nickname that it was a joke question....
[04:38] twittard	heh
[04:38] twittard	I just used the wrong lingo. Networking is not my forte.
[04:38] Pe_Ell	you want to embed the 0mq socket into your application?
[04:39] twittard	Basically, pretend you went "TCPSocket.connect" and it magically opened a connection for you. Then you call ".send" and send it some series of bytes that you cobbled together.
[04:39] twittard	Pe_Ell: well, yeah. I'm currently bargaining with the notion that I might have to write my own networking code.
[04:40] Pe_Ell	you need your application traffic to be distributed?
[04:41] Pe_Ell	or you just need to make a connection between two application processes on different machines?
[04:41] twittard	Yes; however, to someone else's application that has its own way of structuring messages
[04:42] twittard	Pe_Ell: The later, more than the former.
[04:42] Pe_Ell	gonna guess no, but hopefully someone with more knowledge will answer..... :)
[04:43] twittard	my knowledge of 0mq is nil. my education process generally starts with some stupid questions.
[04:44] guido_g	ÃMQ can't talk to arbitrary endpoints, it needs to be a ÃMQ aware endpoint due to the fact that there is "invisible" communication going on on the wire
[04:44] Pe_Ell	fair enough, I do the same thing after I read a few webpages... :)
[05:16] rntz	Is there some way to wait for a socket to send all its queued messages to the network?
[05:17] rntz	analogous in some sense to fflush()
[05:17] guido_g	nope
[05:17] rntz	... really? There's no way to do that? That seems like a terrible oversight.
[05:19] guido_g	Ã¸mq is async in nature, so waiting was not built in
[05:21] sustrik	guido_g, entz: it's a actually a new feature in 2.1
[05:21] guido_g	but only when you close the socket, right?
[05:21] sustrik	sure
[05:22] guido_g	flush on close, so to speak
[05:22] sustrik	otherwise zmq_term hangs :)
[05:22] rntz	"flush when you close the socket" actually covers precisely what I was trying to do
[05:22] guido_g	not a practical way of flushing messages out
[05:22] sustrik	use the master branch from the git repo
[05:22] sustrik	guido_g: yes, right
[05:23] guido_g	but there more importnat things to improve
[05:23] sustrik	?
[05:23] guido_g	like how do i "reset" a req socket if the peer died before an answer was received
[05:24] sustrik	yes, that has to be done
[05:24] guido_g	full ack
[05:24] sustrik	i simply have no resources to do all the should be done :\|
[05:24] guido_g	otherwise the req/rep thingy is not usable the way it should be used
[05:24] guido_g	there might be light
[05:25] sustrik	we also need subscription forwarding etc.
[05:25] guido_g	i took some time of, so it might happen that i look into some of these things
[05:25] sustrik	wow
[05:25] guido_g	of course i'd need a guide
[05:25] sustrik	that would be great
[05:25] sustrik	i'll be happy to help you
[05:25] sustrik	actually
[05:25] sustrik	it may be a good intention to write a "how it works" guide
[05:26] sustrik	which in turn would allow other people to jump in
[05:26] sustrik	incentive*
[05:26] guido_g	uhm... i'd let pieter do the writing for humans :)
[05:26] Pe_Ell	heh
[05:26] sustrik	the problem is he doesn't know how it works internally
[05:27] Pe_Ell	I find his guide amusing
[05:27] guido_g	sigh
[05:27] sustrik	but maybe we can do that together
[05:27] sustrik	it is
[05:27] guido_g	yes, but some parts are really irritating
[05:27] sustrik	i'll give the details, pieter writes the text...
[05:27] sustrik	like?
[05:27] guido_g	like this wholwe mama, papa thingy, i needed a second browser tab so i could look up the definitions
[05:28] sustrik	what's mama, papa?
[05:28] guido_g	see
[05:28] guido_g	ch 3 of the guide
[05:28] sustrik	let me see
[05:28] guido_g	don't... too late...
[05:28] guido_g	:)
[05:29] sustrik	have the quality of the guide deteriorated?
[05:29] sustrik	or is it just that new chapters should not be there?
[05:30] guido_g	the quality is good, imho
[05:30] Pe_Ell	heh
[05:30] sustrik	chapter 1. looks much the same like it did before
[05:30] Pe_Ell	the pages are rather long
[05:31] Pe_Ell	but there's a lot of ideas packed into them
[05:31] guido_g	right
[05:31] Pe_Ell	you added more to page2 and expanded page3 recently
[05:31] sustrik	it was pieter not myself :)
[05:31] guido_g	but given the very short time pieter managed to that written, it's a good thing
[05:31] sustrik	yes, he's actually very good at writing
[05:31] guido_g	*to get
[05:32] sustrik	guido_g: so your prolbme is that it introduces new terminology, right?
[05:33] guido_g	yes
[05:33] guido_g	but no-one else complained so far, so wouldn't take it too serious
[05:33] Pe_Ell	I thought that part was funny, but not sure of a better way to put that that isn't rather technical....
[05:34] sustrik	well, if there's a well-coined term instead of mama and papa
[05:34] sustrik	we should rather use that
[05:34] guido_g	full ack
[05:34] guido_g	but i guess this is the "pieter the sales man" part that's shining through :)
[05:35] sustrik	:)
[05:36] sustrik	but thst doesn't have to affect usability of the guide...
[05:36] guido_g	CH3 "Custom Request-Reply Routing" there it is
[05:37] sustrik	finally scrolled that far :)
[05:38] guido_g	you could have used the table of content at the top :)
[05:38] sustrik	i see
[05:38] Pe_Ell	he's making fun of the old stereo types where the at the dinner table the mom will just talk and the dad only talks when spoken to directly
[05:39] guido_g	might be
[05:39] sustrik	just reading it
[05:39] Pe_Ell	that was how I took it
[05:39] guido_g	but he's using the names all after, so it's confusing for people who are used to req/rep and friends
[05:39] sustrik	can you propose better naming?
[05:40] sustrik	ah
[05:40] guido_g	i knew this was comming...
[05:40] sustrik	they are synonyms for existing socket types
[05:40] sustrik	now i realised that
[05:40] guido_g	i'm actually happy that i'm not the only one who was/is confused
[05:41] sustrik	ok, i'll ask pieter mama & papa as an allegory but stick with official socket type names
[05:41] sustrik	...to use...
[05:41] guido_g	thx
[05:42] Pe_Ell	he explains the names btw...
[05:42] Pe_Ell	you read that right?
[05:42] sustrik	yes, that's fun
[05:42] sustrik	what i am saying is, let's keep that part in
[05:42] guido_g	yes, but if you're used to the socket names you have to learn "additional" names w/o any real meaning in this context
[05:42] sustrik	REQ socket it mama. blah-blah
[05:43] sustrik	but use REQ later on, non mama
[05:43] Pe_Ell	true
[05:43] Pe_Ell	course if you already know the socket names and types... would you be reading that?
[05:43] sustrik	possibly not
[05:43] Pe_Ell	I only read it because I knew next to nothing about how 0mq worked... :)
[05:43] guido_g	i tried to read that part last week after work, but failed because i forgot the menaing of mama and papa and whatnot the next day
[05:44] guido_g	oh, don't get me wrong
[05:44] guido_g	the concepts tought in there are important
[05:44] sustrik	yup, i've got that
[05:44] Pe_Ell	yeah my issues was there was so much to read and I read it over a week and a half after work I'd have to go re-read previous parts since I'd already forgotten the concepts...
[05:44] guido_g	these things will be the bulding blocks to use for apps
[05:45] sustrik	ack
[05:45] Pe_Ell	and I was almost done when he updated the docs again so I had to start over one chapter 2... :)
[05:45] sustrik	i've gave up already :)
[05:45] guido_g	hehe
[05:45] sustrik	wait till it stabilises
[05:45] Pe_Ell	wasn't a big deal for me though... I tend to work on more than one project at a time.. so I'm used to having to crossreference things from memory leakage..
[05:45] guido_g	it's been stable the last week! :)
[05:46] Pe_Ell	hehe
[05:56] guido_g	hmmm... amsterdam
[05:58] sustrik	guido_g: where are you based?
[05:59] guido_g	hamburg, germany
[05:59] sustrik	that's not that far,no?
[05:59] guido_g	my ip is a bit missleading :)
[05:59] Pe_Ell	geoip tends to be very misleading..
[05:59] guido_g	1h flight or 5.5h train
[05:59] sustrik	ah, not that close
[05:59] Pe_Ell	I need to go to Germany... Holand sounds like fun too....
[05:59] guido_g	Pe_Ell: not really, you'll hit the right town quite often
[06:00] sustrik	looks much closer when looking from afar
[06:00] guido_g	hehe
[06:00] Pe_Ell	really? I've found that depending on how dense the area is it won't be very accurate beyond the region for large portions of europe...
[06:01] Pe_Ell	same as the US when you're in places like LA or the bay area in California
[06:01] guido_g	true
[06:01] Pe_Ell	the ISPs all route the traffic out of two or three nodes and those aren't anywhere near the uers
[06:01] guido_g	but in my case it's due to a vpn
[06:01] Pe_Ell	ah
[06:02] sustrik	on the other hand, if there's you, gonzalo, mikko, pieter, myself + 1-2 more guys it may be worth of coming
[06:02] sustrik	let's see whether anyone else joins
[06:02] guido_g	would be cool
[06:03] Pe_Ell	err, less
[06:03] guido_g	i could book it as a "conference" :)
[06:03] sustrik	i haven't been in amsterdam for 20 years or so :(
[06:03] sustrik	definitely, a mini-conference
[08:11] mikko	sustrik: re: jzmq 10
[08:12] mikko	sustrik: i wonder if imatix has capabilities to provide machines to run tests on for language bindings builders?
[08:12] mikko	little and big endians etc
[08:18] pieterh	mikko: it could be good to create some shared virtual machines for this
[08:18] mikko	pieterh: usually they don't have things such as SPARC available
[08:18] pieterh	true
[08:19] pieterh	we do, in fact, have a rather large sparc machine here
[08:19] pieterh	but it would not run all the time
[08:19] mikko	i was wondering if something like hudson would work?
[08:19] mikko	setup master instance and use different architectures as slaves
[08:20] pieterh	looks nice
[08:21] mikko	i can provide x86 (32bit)
[08:21] mikko	i got ~20 DL380s lying around
[08:21] mikko	old ones
[08:21] pieterh	i think boxes is not the real problem
[08:22] pieterh	rather, who would make and maintain such a project
[08:24] mikko	i could make a pilot on some evening
[08:24] mikko	using a couple of virtual machines to see how feasible the idea is
[08:24] mikko	testing with different bingdings and main libzmq
[08:46] sustrik	mikko: what boxes are you actually interested in?
[08:46] sustrik	SPARCs?
[08:47] pieterh	mikko: that would be a good start, then we can extend that over random boxes
[08:48] sustrik	SPARC is good for testing as it's a RISC, pretty different from x86
[08:50] mikko	pieterh, sustrik ill send an email to the list to see what different language bindings would need in terms of testing platform
[08:50] sustrik	good idea
[08:50] pieterh	mikko: sounds good
[08:55] mikko	my experience on anything else than x86(_64?) is very limited so I would assume this kind of setup would help on finding the possible porting issues
[08:55] sustrik	ack
[08:55] mikko	and also zeromq developers get feedback on how changes affect the language bindings
[08:55] mikko	etc
[08:55] mikko	a lot of benefits
[08:58] sustrik	sure, the only problem i see is running such a cluster
[08:58] sustrik	the servers are noisy and power-hungry
[08:59] sustrik	maybe switching them on once a week to do automated builds/tests would work
[09:02] mikko	if the setup is distributed then maybe people with old sparcs or other weird hardware lying around can chip in
[09:08] sustrik	yeah, but i assume the problem is not having the hardware as such, but running it 24/7
[09:08] sustrik	if the setup is distributed you would have then to chase all the participants to turn it on at the same time
[09:08] sustrik	etc.
[09:09] mikko	i mean, there are people who already run hardware 24/7
[09:09] mikko	such as myself
[09:10] mikko	i got a some hardware in a datacentre mainly running idle
[09:10] mikko	hosting my blog and irc client
[09:10] sustrik	right
[09:10] sustrik	iirc mato has an itanium box running in datacenter
[09:10] guido_g	but most people don't have access to a data center
[09:10] mikko	i could just setup new vm instance on the esxi and that would participate in the test system
[09:11] guido_g	so having a box differs from beeing able to run in 24/7
[09:11] mikko	so it would need to be pull instead of push
[09:11] mikko	so that people who turn on the machines occasionally could run the thing and shutdown again
[09:12] guido_g	yep
[09:13] sustrik	which actually makes the design pretty simple
[09:13] sustrik	write the tests
[09:14] sustrik	then the last command in the test stores the results on some 24/7 available box
[09:14] sustrik	thus anyone can run the tests if they feel like it
[09:16] sustrik	i haven't meant just your tests in the core
[09:16] sustrik	the discussion was mainly about testing bindings
[09:16] guido_g	i know
[09:16] sustrik	anyway, the first step: write the tests
[09:16] guido_g	but that doesn't mean that they don't need a lot of love...
[09:17] sustrik	yeah
[09:17] guido_g	yes, then automate the build
[09:17] mikko	i got 21 tests for php bindings this far
[09:17] sustrik	nice
[09:17] mikko	i dont know if other language binding writers do testing
[09:17] sustrik	maybe brian?
[09:18] mikko	http://github.com/zeromq/pyzmq/tree/master/zmq/tests/
[09:18] mikko	yep
[11:38] CIA-20	zeromq2: 03Martin Sustrik 07master * r0bb76b6 10/ src/xrep.cpp : assert when xrep socket gets reconnected in the middle of the shutdown -- fixed - http://bit.ly/cWpYKs
[11:38] CIA-20	zeromq2: 03Martin Sustrik 07master * r2a85cce 10/ src/zmq.cpp : Merge branch 'master' of github.com:zeromq/zeromq2 - http://bit.ly/9SRN5d
[11:55] jasong_at_apache	so right now I'm messing around with zmq to see how it can fit my needs, and I've read a few spots on the mlist about having native secure communication
[11:56] jasong_at_apache	which brings me to that topic, security. right now, where does zmq stand with regard to "in the wild" communication?
[11:56] jasong_at_apache	sorry, thats " secure and in the wild" communication
[12:10] sustrik	it is not ready for that
[12:17] jasong_at_apache	thx
[14:03] CIA-20	zeromq2: 03Martin Sustrik 07master * r1a6cd59 10/ (tests/Makefile.am tests/test_shutdown_stress.cpp): stress test for shutdown process added - http://bit.ly/aTDEaN
[14:23] mato	sustrik: ping
[14:24] sustrik	pong
[14:24] mato	sustrik: i saw you committed a fix to the xrep assert thing
[14:24] sustrik	yes?
[14:24] mato	sustrik: it's the same as the patch you gave me yesterday, right?
[14:24] sustrik	yes
[14:24] mato	ok, good, just checking
[14:25] sustrik	ok
[16:46] CIA-20	zeromq2: 03Martin Sustrik 07master * r2142b89 10/ src/pair.cpp : issue 92 -- Assertion failed: inpipe && outpipe (pair.cpp:86) -- fixed - http://bit.ly/93qQJk
[17:27] mikko	got hudson running and building libzmq for the first time
[17:31] omarkj	Hey guys. I'm publishing data to a zeromq socket with the erlang libs. After a few thousand messages the whole process comes down with "Assertion failed: nbytes == sizeof (command_t) (signaler.cpp:284)"
[17:31] omarkj	Any idea what that could be?
[17:34] omarkj	HWM is at 1.
[18:11] sustrik	mikko: !
[18:12] sustrik	omarkj: what system are you running on?
[18:13] omarkj	sustrik: As in, what OS? In that case, Ubuntu server at the moment. R13B.
[18:13] sustrik	omarkj: can you possibly check what the nbytes value is?
[18:14] omarkj	Hm. I'll try.
[18:14] omarkj	Not quite sure how I'd do that.
[18:14] sustrik	hust modify the code to print it
[18:15] sustrik	if (nbytes != sizeof (command_t) {
[18:15] omarkj	ah, ok
[18:15] sustrik	printf ("%d\n", (int) nbytes);
[18:15] sustrik	zmq_assert (false);
[18:15] sustrik	}
[18:17] omarkj	Hm, I'll have to do this Monday. The test market at the exchange is not running at the moment so I have no data to feed into the process.
[18:18] omarkj	Well, not the levels I need to make it crash.
[18:18] omarkj	Sorry for causing the hassle, but I'll get this value.
[18:18] sustrik	ack, just send the result to the mailing list then
[18:18] omarkj	Okay, thanks a lot.
[18:18] sustrik	np