Tuesday September 21, 2010

[Time] NameMessage
[08:05] CIA-20 zeromq2: 03Martin Sustrik 07master * r9c11886 10/ (src/socket_base.cpp src/socket_base.hpp): The flag in the socket has clear semantics now -- it tracks whether corresponding context was closed, it doesn't track whether zmq_close was called on the socket itself -
[08:05] CIA-20 zeromq2: 03Martin Sustrik 07master * re478468 10/ src/socket_base.hpp : minor error in comment fixed -
[15:56] starkdg hello
[15:57] guido_g hi
[16:12] CIA-20 rbzmq: 03Brian Buchanan 07master * rdc552c2 10/ rbzmq.c : Make PUSH and PULL constants work even if using an older version of the library. -
[16:12] CIA-20 rbzmq: 03Brian Buchanan 07master * rf17f6ca 10/ zmq.gemspec : update gem version to 2.0.9 -
[18:19] AltraMayor Hi there,
[18:20] AltraMayor I've been reading about PNUTS, a NoSQL solution written by Yahoo!
[18:21] AltraMayor A very important element of their design is their message brokers, which guarantee that messages aren't lost. For example, they can make brokers replicate messages with others to support failure.
[18:21] AltraMayor Is this use case supported by zeromq?
[18:21] guido_g no, zeromq doesn't have that
[18:22] guido_g <- more there
[18:22] AltraMayor Thanks, guido_g.
[18:23] AltraMayor I've read that page, but since it says that's more coming, I couldn't assume that it's all.
[18:23] AltraMayor Does anyone know a message broker designed to support that level of failure?
[18:24] guido_g you m ight have skipped the important things
[18:24] AltraMayor I don't follow you, guido_g.
[18:24] guido_g like that ømq doesn't have or is a message broker
[18:25] AltraMayor I understand that zeromq isn't a broker, but its durable option isn't enough.
[18:25] cremes AltraMayor: the roadmap is a wiki page that can be edited by anyone who signs up for an account
[18:25] cremes if you think there are things missing, feel free to add them
[18:25] AltraMayor I also understood I could implement a broker as reliable as I want using zeromq, but this would be just a piece of what I'm looking for.
[18:29] AltraMayor Cremes: what's the link to the roadmap page you mentioned?
[18:31] cremes AltraMayor: it's off the main page, left side under development:
[18:31] AltraMayor Cremes: do you think there's interest among the developers to implement this level of robustness? As I read, zeroqm is implemented by iMatix to meet their demands, I don't know if they'd be interested in a wish list of a guy that wants to have an open-source alternative to PNUTS.
[18:32] cremes good point... i heard their ceo speak on that very topic yesterday
[18:32] cremes if you write the code, it will get done
[18:32] cremes if you pay them to write the code, it will get done
[18:35] AltraMayor I understand, but I'm looking for pieces to estimate the total effort to have an opensource PNUTS.
[18:36] cremes okay, well i don't know what pnuts is :)
[18:39] AltraMayor Cremes: can you provide me a small description how you use 0mq?
[18:41] cremes AltraMayor: sure...
[18:41] cremes i use it to tie together a bunch of distributed components; i used to use amqp but threw it away for 0mq
[18:41] cremes i do some pub/sub and some request/reply style message passing
[18:41] cremes that's the short version
[18:42] AltraMayor How do your components deal with failure? Do you just restart them?
[18:42] cremes it depends
[18:42] cremes if certain components fail, it is fatal for the whole system
[18:42] AltraMayor By the way, PNUTS is a NoSQL that supports replication/accesses/updates among datacenters.
[18:43] cremes if others fail, i detect their absence and just restart/resend their jobs to surviving nodes
[18:43] cremes then perhaps you want to look at mongodb as something similar; 0mq has *nothing* to do with nosql
[18:43] cremes it's not a database in any way, shape or form
[18:47] AltraMayor I see that 0mq isn't a database, but messaging brokers are key in PNUTS' design. Mongodb is the best NoSQL that I know so far, but it don't support datacenters as graceful as PNUTS. And, I'm considering implement some pieces, I just didn't want to start from so low as the message broker.
[18:48] guido_g then get something w/ a broker
[18:49] AltraMayor Ok.
[18:50] AltraMayor Thank you for your time!
[18:50] guido_g best would be broker which stores the messages in a reliable way distributed among datacenters
[18:51] guido_g now wating for the stack overflow...
[20:56] Samy like the new website
[20:56] Samy It would be nice if there was a more obvious link to the source repository.
[20:59] Samy Ah, !=
[21:07] mato Samy: yeah, the .com site is intended as a portal for new users
[21:08] mato Samy: those who just want a first quick look
[21:11] jond mato: hi
[21:14] mato jond: hi there
[21:15] jond mato: i've been looking at the select.cpp and i think there is an issue
[21:15] jond because I'm fixing something similar in the dayjob
[21:17] mato jond: what kind of issue?
[21:17] jond as well as checking that the array doesnt get to FD_SETSIZE, it needs to check that the fd_ in add_fd is actually greater than FD_SETSIZE
[21:18] mato why would the actual fd matter?
[21:18] jond FD_SET, CLR etc don't work on linux above FD_SETSIZE. it's on the man page at the bottom; I had a process with 6000 fd's open and once the sockets started connecting and getting fd > 1024 bad things happen
[21:19] jond fd_set is normally implemented as a bitvector of fixed size
[21:19] mato ah, you're right
[21:19] mato yeah
[21:19] mato good point
[21:20] mato well, the select implementation has not really been stress-tested properly
[21:20] mato it's only there for windows benefit (and vms I guess)
[21:20] jond we raised call with redhat and even if you recompile glibc etc the kernel has it at 1024. My problem is that the 6000 fd's are files opened early and the sockets connect later and get to high fd.
[21:21] jond i'm changing to poll
[21:21] mato well, if you're dealing with 000's of fds you really should not be using select at all :-)
[21:21] jond on solaris I believe you can just redefine FD_SETSIZE
[21:21] mato and poll is not too performant with lots of fds either, you really want an edge-triggered interface
[21:22] mato anyhow, thanks for the tip; i'll make a note to make a patch that adds in the appropriate assertions
[21:23] jond yep I know, but I 'inherited' this code; we only have a 600 tcp connections, but open lots of files....
[21:24] jond mato: edge-triggered, you mean epoll?
[21:27] mato jond: yes
[21:27] mato jond: unfortunately epoll never got standardised so is available only on Linux
[21:27] mato jond: *BSD has kqueue, Solaris has Event Completion
[21:27] jond i'll take a look tommorrow; thanks for the tip;
[21:27] mato jond: (re the Solaris implementation, see
[21:28] jond mato: cheers
[21:36] bgranger mato: did you see my reply about the message related deadlock you were seeing?
[21:38] jond bgranger: i did. is there a link to that pyzmq branch
[21:38] bgranger Sure, right here...
[21:38] dermoth Is there any example of XREQ/XREP out there? I've used REP/REP successfully in the past and now I'd like to add devices in the middle, and I believe I need XREQ/XREP for that.
[21:39] bgranger
[21:39] bgranger Here is the class
[21:39] bgranger
[21:40] bgranger We are still doing code review so a few more things might change in the API, but I think things are mostly in place.
[21:40] mato bgranger: hi
[21:41] bgranger hi
[21:41] mato bgranger: yes, i saw that, am travelling most of this week (Linux Kongress in Nurnberg, Martin Sustrik is presenting)
[21:41] bgranger Nice
[21:41] jond bgranger: thanks, i'll take a look. I think that function that's added in py2.7 for PendingCall looks interesting and could still provide a version which doesnt need the MesssageTrcaker
[21:41] mato bgranger: will look at it next week once I get back home
[21:41] bgranger Great, sounds good, by then it will probably be in our master.
[21:41] mato bgranger: can't really test it reliably while on the road.
[21:41] bgranger OK
[21:54] cremes dermoth: hopefully this will help a little:
[21:58] dermoth [bbl]
[23:27] tylergillies whats the difference between req/rep and upstream/downstream?
[23:31] vanadium req/rep are bidirectional/back and forth, upstream/downstream are push/pull and one of them always sends and the other always receives