Wednesday July 27, 2011

[Time] NameMessage
[07:30] CIA-32 libzmq: 03Martin Sustrik 07vtcp * r588c728 10/ src/vtcp_connecter.cpp : vtcp_connecter fixed ...
[14:33] Seta00 ;)
[15:15] kilpatds S'ok. I'm about to expose my stupid. :)
[15:16] kilpatds I've got a ROUTER socket. I'm using zmq_poll. What does ZMQ_POLLOUT mean? That there exists at least one client that I can write to? Or that there does not exist a single client that I can't write to?
[15:17] traviscline imo if em-zeromq and the like is on the labs page then maybe should be as well
[15:20] kilpatds (also: says that if the first write succeeded, all others will. Up to available memory, then?
[15:20] kilpatds What should I do if I have 10G of data on disk I'm trying to pump to a client. There's no need to write faster than the client can read, so I'd really like some feedback mechanism to tell me to slow down. Is the assumed solution "chunk your steam, and use a protocol to send chuncks"?)
[15:22] sustrik with ROUTER socket, POLLOUT is always true
[15:22] kilpatds Thanks. Good to know
[15:50] xyzzy Alright, I'm not seeing it in the guide, and I'm supposing the problem is my expectation of the command. In the C# wrapper there's a RecvAll() function. My expectations where it'd return a queue of published updates for me to go through on the clientside. However, everytime I run it, I still only get one publish. Is that how it's supposed to work? How would I make it match my expectations?
[15:53] xyzzy and exposing my stupid already, I mean in the c# language bindings. :p
[15:53] mikko xyzzy_: it probably returns all parts of a multipart message
[15:53] mikko i would imagine
[15:54] mikko you could easily test that by publishing a multipart message
[15:58] xyzzy I'll go test that and see if it behaves differently, wasn't even thinking that route. You're quite likely right.
[16:38] cremes wow, after all this time now i have a question!
[16:39] cremes for a PUSH socket with a HWM set to 1, send should block when all PULL sockets have 1 message in their queue
[16:39] cremes (yes, there might be some in system buffers, etc.. let's ignore that for now)
[16:39] cremes calling zmq_send() on that socket should block when HWM is reached
[16:40] cremes what should happen when calling zmq_send() with ZM_NOBLOCK set? it should
[16:40] cremes return EAGAIN, right?
[16:50] sustrik cremes: yes
[17:23] cremes are there still planned fixes for the 2.1.x branch?
[17:23] cremes i discovered that PUSH/PULL balancing does not work properly on windows
[17:23] cremes for version 2.1.7
[17:43] sustrik cremes: definitely report the issue
[17:43] cremes sustrik: i'm reading through the new docs on how to do so...
[17:43] cremes i have recreated it with my ruby binding so i need to translate to C
[17:43] cremes this might take me a little bit
[17:44] Seta00 ha, can you post your ruby code cremes?
[17:44] cremes Seta00: sure, give me a second
[17:58] cremes Seta00: here's a gist with the code & output for both osx & win7
[17:59] cremes github reordered the files for some reason...
[17:59] Seta00 it always does that :/
[17:59] Seta00 pretty lame
[17:59] cremes you can see that on windows, a send(ZM_NOBLOCK) *always* succeeds even when HWM should have been hit
[17:59] Seta00 you never sent a link
[17:59] cremes it just round-robin delivers the messages to any PULL sockets
[17:59] cremes
[17:59] cremes :)
[18:00] cremes also, copy/paste from powershell f'ed up the line wrap <sigh>
[18:01] cremes so, i don't know if this fails because HWM isn't respected on windows (send should return EAGAIN)
[18:01] Seta00 weird!
[18:01] cremes or if there is a deeper issue
[18:01] cremes i'm 99% sure this is a PUSH problem though... if i run the sender on osx
[18:02] cremes and the clients on windows, it all works as expected
[18:08] xyzzy mikko: After lots of fiddling and headaches, you're exactly right. When sending a multipart message RecvAll splits the entire message into a queue structure :) Not exactly what I wanted, but, at least now I know how it works. Thanks!
[18:11] sustrik cremes: why do you believe it should block?
[18:13] cremes sustrik: if the client is busy doing some work while the sender tries blasting it
[18:13] cremes with messages, the HWM should be hit very quickly
[18:13] cremes the osx output shows that
[18:13] cremes if HWM is reached, a NOBLOCK send should return EAGAIN
[18:14] sustrik depends on how fast is the publisher able to push messages to the wire
[18:14] cremes (which in my output is listed as false)
[18:14] sustrik if it's able to keep pace with the publishing rate
[18:14] sustrik the messages just flow through the PUSH socket
[18:14] cremes then HWM has no effect for PUSH sockets?
[18:15] sustrik HWM is amount of messages stored in the socket
[18:15] sustrik as you call zmq_send() there's one message stored
[18:15] cremes right, in this case it is in the PUSH socket; the PULL sockets are only trying to read from their
[18:15] sustrik as it gets sent to the wire
[18:15] sustrik it's de-queued
[18:15] cremes queue every 1 or 2 seconds
[18:16] sustrik there are 4 buffers on the way though:
[18:16] cremes so a slow PULL client can't exert any back-pressure on a PUSH socket?
[18:16] sustrik yes it can
[18:16] sustrik but all the 4 buffers have to be filled
[18:16] cremes ok, i'm going to wait for your entire explanation :)
[18:16] sustrik these are the buffers:
[18:17] sustrik 1. PUSH's message queue
[18:17] sustrik 2. PUSH's TCP tx buffer
[18:17] sustrik 3. PULL's TCP rx buffer
[18:17] sustrik 4. PULL's message queue
[18:17] sustrik the backpressure has to propagate through all of there
[18:17] sustrik these
[18:18] sustrik so, if client doesn't read messages
[18:18] sustrik it's (PULL's) message queue fills up
[18:18] sustrik when HWM is reached is stops reading messages from the TCP socket
[18:18] cremes i'm going to test this on windows using a *large* message that should fill any buffer and see what happens
[18:19] sustrik thus the TCP rx buffer starts filling up
[18:19] sustrik etc.
[18:20] sustrik cremes: also, afaics the receiving side doesn't set HWM
[18:20] cremes right
[18:20] sustrik so any number of messages can be stored in the PULL socket
[18:21] sustrik so what may be happening is that all the messages are passed to the PULL's queue
[18:21] sustrik and there's no HWM so backpressure is never exercised
[18:22] cremes sustrik: seems odd that on osx it behaves as i expect (back pressure almost immediately) while
[18:23] xyzzy This doesn't seem to be in any of the C# guides... but it solved a lot of my issues. Making sure to check that socket.GetSockOpt(SocketOpt.RCVBUF) > 0 before attempting to read. Maybe that's glaringly obvious to most... but, yeaaaah :P not me :P
[18:23] cremes on windows it is able to send as fast as it can
[18:23] cremes sustrik: btw, just tested with an 8k message and it still behaved oddly on windows (no EAGAIN)
[18:23] sustrik try setting HWM on the pull socket
[18:24] sustrik xyzzy_: RCVBUF=0
[18:24] sustrik ?
[18:24] sustrik how come
[18:24] sustrik what OS are you using?
[18:24] xyzzy If it's > not =
[18:24] pieter_hintjens cremes: you sure you're setting the HWM before connect/bind?
[18:24] cremes pieter_hintjens: yes, first thing i checked :)
[18:25] pieter_hintjens nick pieterh
[18:25] sustrik xyzzy_: RCVBUF is the size of TCP rx buffer
[18:25] sustrik it should never be zero
[18:25] xyzzy Wow :/ That's counterintuitive and voodoo then :P
[18:26] sustrik can you explain what's going on?
[18:26] pieterh xyzzy_: ? you're comparing oranges and chickens here...
[18:26] cremes each message i'm sending via PUSH is now about a megabyte
[18:26] xyzzy I have a queue, I assign client.RecvAll to it, with Noblock. If there's nothing being sent, it errors. So I have a try/catch block
[18:26] cremes setting HWM on PULL had no effect
[18:26] xyzzy Without the check on RCVBUF, the catch block activates everytime the loop goes through and there's no data to be recv'd.
[18:26] sustrik cremes: then is seems like a bug
[18:26] pieterh xyzzy_: use poll
[18:26] xyzzy When I added the check. It magically stopped.
[18:27] xyzzy So, I thought my experiment was right.
[18:27] pieterh xyzzy_: there's no magic in software
[18:27] xyzzy but, what I'm hearing is that has nothing to do with something being ready to read from the socket xD.
[18:27] pieterh checking rcvbuf size > 0 is like testing 1 > 0
[18:27] sustrik xyzzy_: i am not familiar with C# binding
[18:27] sustrik but isn't the error just EAGAIN?
[18:27] cremes sustrik: i'll open an issue... it may take me a few days to get a C repro written... i don't have a
[18:27] cremes simple dev environment on windows
[18:28] sustrik ie. "try to recv later on"
[18:29] xyzzy pieterh, I know there's really no magic, it's trying to figure out why the code experiment suddenly worked after it that's doing something odd :P
[18:29] pieterh xyzzy_: can you make a _minimal_ test case and post it somewhere?
[18:29] sustrik the standard behaviour with NOBLOCK is either get a message or get an EAGAIN error
[18:30] pieterh questions like this, without code to look at, are unanswerable
[18:30] xyzzy pieterh, there's not a test case, it's me breaking my brain trying to get a grasp of things. The minimal case is knowing when to read and only storing something when there's data to be had.
[18:30] xyzzy I suppose minimal is
[18:30] xyzzy string blah = socket.recv(encoding.unicode);
[18:31] pieterh xyzzy_: not here, I mean working complete minimal test case
[18:31] rpgdude is there a way to build a message queue where the client doesn't expect a reply
[18:31] xyzzy actualy, with NOBLOCK on that too. It erros if there's nothing to recieve. I was trying to figure out a way to avoid that. :P
[18:31] sustrik what's the error?
[18:31] sustrik rpgdud: either PUB/SUB or PUSH/PULL
[18:32] xyzzy byte array cannot be null
[18:32] sustrik heh
[18:32] sustrik sounds like a bug in C# binding
[18:32] rpgdude what works better with worker threads?
[18:32] xyzzy The suggestion to use polling is probably correct, but, I haven't looked at any of those examples.
[18:32] xyzzy So, looking there and will see what happens.
[18:33] sustrik recv is called with NOBLOCK, zmq_recv() returns EAGAIN, the C# binding still tries to return a message and fails in the process
[18:34] sustrik rpgdude: depends on the use case, check semantics of individual socket types in zmq_socket(3) man page
[18:36] pieterh xyzzy_: plz read the guide before asking questions here
[18:36] pieterh it will help you a lot in understanding 0MQ
[18:37] sustrik C# implementation of recv() is pretty weird
[18:37] sustrik
[18:38] sustrik is seems to enter a busy loop in case NOBLOCK is specified and EAGAIN is returned
[18:38] Seta00 yeah :|
[18:38] Seta00 took me a while to find out why my app was using 100% CPU
[18:39] pieterh huh
[18:39] sustrik why not open an issue on clrzmq2?
[18:40] Seta00 don't know, really, I dropped C# soon after that problem, forgot about it
[18:40] sustrik anyway, it should not produce the error xyzzy_ had got
[18:41] sustrik xyzzy_: aren't you sending a 0-byte message btw?
[18:41] sustrik the error seems to imply that C# doesn't allow for 0-byte arrays
[18:41] sustrik whereas 0MQ allows for 0-byte messages
[18:45] neale1 quit
[18:45] neale1 [sorry wrong window!]
[18:47] xyzzy pieterh, I am reading the guide but the c# code samples dont document some of the things it does, so it requires lots of guess work
[18:48] pieterh xyzzy_: seems the C# binding is a little eccentric sometimes... may need cleaning up
[18:48] pieterh if you can document any issues with it, that'd be helpful
[18:49] xyzzy I will try, I an still very newbish though.
[18:50] xyzzy sustrik, the client may or may not have sent data. don't know at that point.
[18:50] sustrik try sending at least 1 byte in the message
[18:54] xyzzy When the client send data out does send more than 1 byte. At that point it isnt known whether or not the client has sent anything.
[18:59] xyzzy I suspect the earlier suggestuon of polling is probably what I need to look at.
[19:01] sustrik xyzzy_: yes, that should help
[19:01] sustrik but there's definitely a bug in C# binding
[19:02] sustrik so i would suggest that you report it so that it can be fixed
[19:02] xyzzy I am glad my ignorance helped reveal something, but, not sure what that something is..
[19:04] sustrik just post your code to the issue and mention what error you are getting
[19:06] xyzzy alright :)
[20:38] cremes pieterh: ping
[20:38] pieterh cremes: hi!
[20:38] cremes i have a C program to reproduce my PUSH/PULL issue but i am having trouble compiling on windows
[20:38] cremes (i'm missing pthread.h)
[20:38] pieterh yeah, doesn't exist on win32
[20:38] cremes any chance i can send you a copy and you can check it out?
[20:38] pieterh best I can advise is to use czmq for that
[20:38] cremes or should i just split the test into client/server pieces?
[20:39] pieterh it has a thread api
[20:39] cremes um... ok
[20:39] pieterh send me the source & I'll make it work
[20:39] pieterh i assume the problem isn't win32 specific
[20:39] pieterh you have an issue number for this?
[20:40] cremes
[20:40] cremes i forked the issues repository
[20:40] cremes this code is the repro for an issue i haven't opened yet... i wanted to make sure
[20:40] cremes the code illustrated the problem first :)
[20:40] pieterh cremes: ah, good... send me a pull request for the issues repo
[20:40] cremes roger
[20:40] pieterh open an issue, if it's not a real problem we'll close it again
[20:50] cremes pieterh: issue 235 is opened
[20:51] cremes pieterh: i'm part of the zeromq org, so i pushed directly to the issues repository (no pull request required)
[20:51] pieterh cremes: nice, I'll do this tomorrow
[20:51] cremes awesome, thanks
[21:10] Seta00 cremes: install pthreads-win32
[21:30] pieterh Seta00: hardly worth it for a test case
[22:43] jhawk28 what is the best way to update the Java examples in the Guide? The Poller examples are wrong
[22:43] jhawk28 should I fork and provide a pull request?
[23:42] Seta00 jhawk28: yeah