IRC Log

Tuesday October 4, 2011

[Time] NameMessage
[07:34] CIA-79 libzmq: 03Steven McCoy 07master * rcc5d316 10/ src/pgm_socket.cpp : Re-add PGM rate limiting ...
[07:41] CIA-79 libzmq: 03Steven McCoy 07master * rf5f1a8b 10/ (3 files in 2 dirs): New upstream OpenPGM maintenance release 5.1.118. ...
[08:28] mikko good morning
[08:30] sustrik morning
[09:02] mikko sustrik: did you see the bug i opened last night?
[09:04] sustrik yes
[09:05] sustrik haven't inspected it in detail yet though
[09:18] mikko sustrik: it seems odd that i can send 102064 messages with hwm 1
[09:18] mikko especially when noone reads
[09:18] sustrik yes
[09:19] sustrik some are in TCP buffers though
[09:33] mikko sustrik: might be majority
[09:34] mikko this might be why i added the devices to pzq earlier
[09:34] mikko it allows better hwm management (over inproc)
[09:35] mikko previously i remember that two messages or so were pushed to pipe
[09:35] mikko now i see ~5k or so
[09:35] mikko not sure if those just go to buffers
[09:35] mikko if this is expected behaviour then i need to rething the logic
[09:56] sustrik mikko: what you can do is to use the acks for flow control
[09:57] sustrik ie. don't send more messages if there are more then N messages on the fly
[09:59] mikko yes, that's what i am thinking
[09:59] mikko that is very prone to slow acks though
[10:01] sustrik what's the problem with that?
[10:01] sustrik that's what is wanted, no?
[10:01] sustrik ie. if acks are slow the message flow is slow as well
[10:01] mikko that is not per peer limitation
[10:01] sustrik i see
[10:02] sustrik then you'll need a ROUTER socket instead of dealer
[10:02] mikko yes
[10:02] mikko need to change the flow a bit i guess
[10:06] mikko i guess with router i need to maintain more state
[10:06] mikko that is the downside
[12:01] beobal hi, i'm having a small problem getting to grips with zmq_poll - specifically I'm using the java bindings
[12:01] beobal Poller.poll always seems to return immediately, regardless of any timeout value
[12:02] beobal I understand the reasoning behind http://lists.zeromq.org/pipermail/zeromq-dev/2010-June/003887.html
[12:02] beobal but a timeout of -1 isnt working for me either
[12:02] beobal am i right to think that Poller.poll(-1) should block until event(s) occur?
[12:14] mikko beobal: yes
[12:23] beobal ok, i think its a threading problem
[12:24] beobal I'm calling poll from 1 thread thats in a loop, but the sockets are created by a different thread
[12:24] beobal they share a context and the polling thread owns the poller, but it didn't create the sockets. could that be the cause?
[12:24] mikko you should access sockets only from one thread
[12:25] mikko you need to do full memory barrier before using socket from another thread
[12:25] mikko (mutex or so)
[12:25] beobal i only call socket.recv from the poller thread, the other thread is just responsible for creating the socket
[12:25] guido_g whcih is *not* the intended way to use ømq sockets
[12:26] guido_g why do you create sockets in a diferent thread?
[12:27] beobal I was doing that because the main thread is busy in the poll loop. would the right way to do it be to add another socket to the poller and use that to signal to the poller thread that a new socket needs creating?
[12:28] guido_g for example
[12:29] mikko that seems like very odd design
[12:29] mikko without knowing more about it
[12:30] guido_g having a control socket is not odd per se
[12:30] guido_g or did i get your comment wrong?
[12:31] beobal i'm thinking that i'll will have potentially 1000s of sockets, and don't want to spawn a thread to manage each one (calling recv). the idea was that 1 thread owns the poller and sits in a loop polling it. then new sockets are created as clients connect, and get registered with the poller
[12:32] beobal it seems to work as I expected, except for the timeout behaviour of the poller
[12:32] beobal but i can see why that is now
[12:33] mikko hmm
[12:33] mikko you have one socket per client?
[12:33] beobal yes
[12:33] mikko why is that?
[12:34] beobal the sockets of SUB type, and each client is subscribing to a unique channel
[12:34] beobal i could do that with a single socket i guess, and use filtering
[12:35] mikko do the clients subscribe to just one channel?
[12:35] beobal yes
[12:35] guido_g ok, now it is odd
[12:35] beobal lol
[12:35] mikko beobal: why do you use pub/sub in that case?
[12:35] mikko why not ROUTER socket and just route the messages to correct clients?
[12:35] beobal sorry, they're actually PULL
[12:35] mikko that way you could use one socket
[12:36] mikko you could have let's say a pool of worker threads that get fed through inproc socket
[12:36] mikko and one router socket facing the clients
[12:37] mikko client --- tcp --> ROUTER --- inproc / load balance --> worker threads
[12:38] guido_g if it has to be threads, might also be processes (even on other machines)
[12:38] beobal guido_g: its actually scala actors, so backed by a thread pool, but yes I see your point
[12:38] mikko yes, i would use threads if local
[12:39] guido_g scala? asctor? hear, hear! :)
[12:39] mikko beobal: there are several benefits to this design
[12:39] mikko beobal: one of which is a lot less configuration on clients
[12:39] mikko they need to know about single or a few endpoints
[12:39] mikko rather than each one their own
[12:40] beobal cool, thanks - i'll go and read up on ROUTER sockets then
[12:40] beobal thanks for the help
[12:41] mikko no problem
[12:41] guido_g you also might want to have a look at the guide, at least for ideas on what is possible
[12:42] beobal cheers guido_g, will do (again) :)
[12:42] calvin is there a particular reason why creating sockets on one thread and then calling recv on another (although the socket would never be calling recv on multiple threads, just one for the duration of the application) is bad?
[12:42] calvin i understand it's not the intended use case, but i've done it in the past with a lot of luck and am curious what the implications are in case i'm looking at a ticking time bomb
[12:45] mikko calvin: there is no issue here as long as you guarantee a full memory barrier before using on another thread
[12:46] calvin what do you mean by a full memory barrier? I know what a memory barrier means atomically
[12:46] mikko http://en.wikipedia.org/wiki/Memory_barrier#An_illustrative_example
[12:46] calvin basically 3 sockets are created on thread A, then socket 1 is used on thread B, socket 2 is used on thread C and socket 3 is used on thread D
[12:46] calvin ok i'll give that a read
[12:47] mikko calvin: if you use mutex or a semaphore before accessing the socket you should be fine
[12:48] mikko calvin: as far as i understand (which isnt much) there might be issues with out of order execution there
[12:48] mikko very short race condition
[12:49] calvin oh interesting
[19:48] larsks Hello all. I'm working with the 0MQ python bindings, and I can
[19:48] larsks (whoops)
[19:48] larsks can't get send_multipart() to operate with zmq.NONBLOCK.
[19:48] larsks (For a PUSH socket, in the case that the corresponding PULL socket is not available/not listening)
[19:49] larsks Should this work?
[20:01] cremes larsks: you need to put your code up on pastie.org or some other paste service so people can look at it
[20:01] larsks https://gist.github.com/1262602
[20:02] larsks It's trivial, which is why I didn't link it initially.
[20:02] cremes and where is the receiver?
[20:02] cremes oh, i see your comment in there
[20:02] larsks I want it to fail (instead of blocking) when the receiver is not available.
[20:02] cremes that's not how 0mq works
[20:02] cremes have you read the guide yet?
[20:03] larsks Indeed.
[20:03] cremes indeed you read it, or indeed you did not? :)
[20:04] larsks I have read through the guide. I would like a failure mode other than blocking for this particular situation. I don't thinkt he PUB/SUB model is appropriate here, and I don't need the request/reply model, either.
[20:04] larsks I'm looking for a one-way message delivery, which is why I'm looking at PUSH/POLL sockets.
[20:04] larsks It works great, other than the blocking behavior.
[20:04] cremes well, the issue is that you are conflating two activities with one socket
[20:04] cremes the push socket cannot tell you that there are 0 pull sockets listening
[20:05] cremes that information is not exposed in the 2.1.x or 3.x apis
[20:05] cremes to make sure someone is listening, your consumers need to heartbeat with another pair of sockets
[20:05] larsks Huh. I figured that internally it would know it had been unable to establish a tcp connection. Perhaps I am misunderstanding the purpsoe of the NONBLOCK flag.
[20:05] cremes probably
[20:06] larsks So if I want to be able to send a message to a receiver *that may not be listening*, what's the correct model?
[20:06] larsks This is "informative" data; it's okay to drop it and it doesn't need to be synchronously delivered.
[20:07] cremes pub/sub is appropriate here
[20:08] cremes think of a PUB socket as a radio broadcaster
[20:08] larsks I thought that PUB wouldn't send without an a priori subscription.
[20:08] cremes if no one is listening, the packets are dropped
[20:08] cremes look at the man page for zmq_socket where all sockets are described
[20:08] larsks More importantly, I though that PUB sockets had to bind() and SUB sockets had to connect() , which is the opposite of what I need.
[20:09] cremes you can see that PUB will drop packets when HWM is exceeded, so just set HWM to 1
[20:09] cremes no, that isn't true
[20:09] minrk all sockets can both bind and connect
[20:09] cremes where are you getting this?
[20:09] larsks All the examples I've seen show a subscriber connecting to a publisher (via connect()).
[20:09] larsks I'll take a closer look.
[20:10] cremes sure, but those are just examples
[20:10] minrk zeromq is entirely agnostic about connection direction - you should be able to toggle bind/connect in every example with ~ no change
[20:11] larsks I had tried this earlier but ran into problems and just figured I was misusing things. I'll give it another try.
[20:12] larsks Thanks for your thoughts.
[20:14] larsks ...nope, I'm still seeing blocking behavior with PUB sockets, too. I've updated https://gist.github.com/1262602 to create a PUB socket, and it still blocks in `send_multipart` if the SUB socket isn't available. As soon as the listener comes up, the message is delivered and send_multipart() unblocks.
[20:14] cremes larsks: set HWM to 1
[20:14] cremes before the bind/connect call
[20:16] minrk connect is blocking, not send
[20:17] minrk add a print in between to verify
[20:17] larsks The connect() call completes successfully. It's blocking in send_multipart, although perhaps under the hood the connection operating is delayed untilt he data is first sent. I'm trying with HWM=1 right now.
[20:17] larsks I have added a print to verify. COntrol flow is definately blocking in send_multipart.
[20:17] minrk ok
[20:18] larsks I have added a print to verify. COntrol flow is definately blocking in send_multipart.
[20:18] larsks (sorry)
[20:19] larsks Setting HWM=1 (s.setsockopt(zmq.HWM, 1)) doesn't appear to affect the behavior, either.
[20:19] larsks I'm going to punt for now and spend some more quality time with the documentation this evening.
[20:19] larsks Thanks again for the suggestions.
[20:23] minrk actually, print statements indicate that it's actually close() that's blocking, not send
[20:23] guido_g your program is blocking after the send
[20:23] guido_g right
[20:23] minrk send completes just fine
[20:23] minrk but close is waiting to flush messages
[20:23] minrk see LINGER sockopt
[20:23] larsks Ah, interesting. I'll try that.
[20:24] guido_g add: s.setsockopt(zmq.LINGER, 0) before the connect and see
[20:24] larsks Hey, look, it works. minrk, thanks for that pointer.
[20:24] minrk print-statement debugging to the rescue!
[20:25] larsks ...although I note that with LINGER == 0, it doesn't actually deliver the messages when the listener is available. It looks like LINGER=1 works, though.
[20:26] mikko good evening
[20:26] minrk In your code, close is called immediately after send_multipart
[20:26] larsks Yeah, it's a timing issue.
[20:26] minrk (it is called automatically on gc)
[20:27] mikko minrk: you remember the issue from other evening?
[20:27] minrk mikko: which?
[20:27] mikko minrk: lost messages on close
[20:27] mikko it was actually a bug
[20:28] mikko on better news, new build hardware is installed
[20:28] minrk excellent!
[20:28] mikko and i should be getting access soon
[20:28] mikko then just need to install vms and we are set to go
[20:29] mikko i think i have some time off next week
[20:29] mikko and been thinking about this snapshot portal
[20:29] mikko where build artifacts could be downloaded easily
[20:30] minrk that would be greatly appreciated (especially by Windows folks, I imagine)
[20:30] mikko do you have automated packaging mechanism in pyzmq?
[20:30] mikko because i guess thats fairly easy to add to listings
[20:30] minrk I don't, I do it all by hand
[20:39] minrk shouldn't be hard, I imagine. The main thing is it should use libzmq built with MSC 1500 (VS 2008), to match Python.
[22:28] adymitruk hi everyone
[22:29] adymitruk anyone here successfully used zeromq in .NET?
[22:50] adymitruk I'm having issues finding the dlls for C#
[22:57] adymitruk anybody here? Do I have to build zeroMQ myself or are there dlls that I can DL somewhere?
[23:04] adymitruk anyone alive in here?
[23:08] mikko adymitruk: a lot of people are on EU timezone
[23:08] mikko s
[23:08] adymitruk I figured :)
[23:09] adymitruk do I need to build from source?
[23:09] mikko http://lists.zeromq.org/pipermail/zeromq-dev/2011-October/013595.html
[23:09] adymitruk or can I find an DLL set to download
[23:09] adymitruk sweet
[23:09] mikko you probably want 2.1.10 with clrzmq
[23:09] mikko also, there are NuGet packages available
[23:09] mikko http://nuget.org/List/Packages/clrzmq2
[23:10] adymitruk no nuget for me
[23:10] adymitruk but you wouldn't use 3.0?
[23:10] mikko it's still in beta
[23:11] mikko unless you need specific features from it, then no
[23:11] adymitruk ok.. I'll do the 2.1 then
[23:12] adymitruk how come there is no link to those on the download page?
[23:12] adymitruk seem's only the source
[23:12] mikko we don't currently do official windows binaries
[23:13] mikko hopefully there is going to be windows snapshot builds for 32/64 soon
[23:16] adymitruk ok.. in that archive i found msvcp100.dll
[23:16] adymitruk is that all I need?
[23:17] adymitruk I found libzmq.dll
[23:17] adymitruk that must be it :)
[23:52] CIA-79 pyzmq: 03MinRK 07master * r7e099c8 10/ (4 files): fix missing ENOTSOCK const ...
[23:52] CIA-79 pyzmq: 03MinRK 07master * rb5d816c 10/ zmq/utils/zmq_compat.h : fix FD_T (again) on x64 Windows - http://git.io/vkNJ3g

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy