[Time] Name | Message |
[08:05] CIA-20
|
zeromq2: 03Martin Sustrik 07master * r9c11886 10/ (src/socket_base.cpp src/socket_base.hpp): The flag in the socket has clear semantics now -- it tracks whether corresponding context was closed, it doesn't track whether zmq_close was called on the socket itself - http://bit.ly/d6ktZI
|
[08:05] CIA-20
|
zeromq2: 03Martin Sustrik 07master * re478468 10/ src/socket_base.hpp : minor error in comment fixed - http://bit.ly/a4RuQJ
|
[15:56] starkdg
|
hello
|
[15:57] guido_g
|
hi
|
[16:12] CIA-20
|
rbzmq: 03Brian Buchanan 07master * rdc552c2 10/ rbzmq.c : Make PUSH and PULL constants work even if using an older version of the library. - http://bit.ly/9ZisAu
|
[16:12] CIA-20
|
rbzmq: 03Brian Buchanan 07master * rf17f6ca 10/ zmq.gemspec : update gem version to 2.0.9 - http://bit.ly/b4wU0A
|
[18:19] AltraMayor
|
Hi there,
|
[18:20] AltraMayor
|
I've been reading about PNUTS, a NoSQL solution written by Yahoo!
|
[18:21] AltraMayor
|
A very important element of their design is their message brokers, which guarantee that messages aren't lost. For example, they can make brokers replicate messages with others to support failure.
|
[18:21] AltraMayor
|
Is this use case supported by zeromq?
|
[18:21] guido_g
|
no, zeromq doesn't have that
|
[18:22] guido_g
|
http://zguide.zeromq.org/chapter:1 <- more there
|
[18:22] AltraMayor
|
Thanks, guido_g.
|
[18:23] AltraMayor
|
I've read that page, but since it says that's more coming, I couldn't assume that it's all.
|
[18:23] AltraMayor
|
Does anyone know a message broker designed to support that level of failure?
|
[18:24] guido_g
|
you m ight have skipped the important things
|
[18:24] AltraMayor
|
I don't follow you, guido_g.
|
[18:24] guido_g
|
like that ømq doesn't have or is a message broker
|
[18:25] AltraMayor
|
I understand that zeromq isn't a broker, but its durable option isn't enough.
|
[18:25] cremes
|
AltraMayor: the roadmap is a wiki page that can be edited by anyone who signs up for an account
|
[18:25] cremes
|
if you think there are things missing, feel free to add them
|
[18:25] AltraMayor
|
I also understood I could implement a broker as reliable as I want using zeromq, but this would be just a piece of what I'm looking for.
|
[18:29] AltraMayor
|
Cremes: what's the link to the roadmap page you mentioned?
|
[18:31] cremes
|
AltraMayor: it's off the main page, left side under development: http://www.zeromq.org/docs:3-0
|
[18:31] AltraMayor
|
Cremes: do you think there's interest among the developers to implement this level of robustness? As I read, zeroqm is implemented by iMatix to meet their demands, I don't know if they'd be interested in a wish list of a guy that wants to have an open-source alternative to PNUTS.
|
[18:32] cremes
|
good point... i heard their ceo speak on that very topic yesterday
|
[18:32] cremes
|
if you write the code, it will get done
|
[18:32] cremes
|
if you pay them to write the code, it will get done
|
[18:35] AltraMayor
|
I understand, but I'm looking for pieces to estimate the total effort to have an opensource PNUTS.
|
[18:36] cremes
|
okay, well i don't know what pnuts is :)
|
[18:39] AltraMayor
|
Cremes: can you provide me a small description how you use 0mq?
|
[18:41] cremes
|
AltraMayor: sure...
|
[18:41] cremes
|
i use it to tie together a bunch of distributed components; i used to use amqp but threw it away for 0mq
|
[18:41] cremes
|
i do some pub/sub and some request/reply style message passing
|
[18:41] cremes
|
that's the short version
|
[18:42] AltraMayor
|
How do your components deal with failure? Do you just restart them?
|
[18:42] cremes
|
it depends
|
[18:42] cremes
|
if certain components fail, it is fatal for the whole system
|
[18:42] AltraMayor
|
By the way, PNUTS is a NoSQL that supports replication/accesses/updates among datacenters.
|
[18:43] cremes
|
if others fail, i detect their absence and just restart/resend their jobs to surviving nodes
|
[18:43] cremes
|
then perhaps you want to look at mongodb as something similar; 0mq has *nothing* to do with nosql
|
[18:43] cremes
|
it's not a database in any way, shape or form
|
[18:47] AltraMayor
|
I see that 0mq isn't a database, but messaging brokers are key in PNUTS' design. Mongodb is the best NoSQL that I know so far, but it don't support datacenters as graceful as PNUTS. And, I'm considering implement some pieces, I just didn't want to start from so low as the message broker.
|
[18:48] guido_g
|
then get something w/ a broker
|
[18:49] AltraMayor
|
Ok.
|
[18:50] AltraMayor
|
Thank you for your time!
|
[18:50] guido_g
|
best would be broker which stores the messages in a reliable way distributed among datacenters
|
[18:51] guido_g
|
now wating for the stack overflow...
|
[20:56] Samy
|
like the new website
|
[20:56] Samy
|
It would be nice if there was a more obvious link to the source repository.
|
[20:59] Samy
|
Ah, zeromq.com != zeromq.org
|
[21:07] mato
|
Samy: yeah, the .com site is intended as a portal for new users
|
[21:08] mato
|
Samy: those who just want a first quick look
|
[21:11] jond
|
mato: hi
|
[21:14] mato
|
jond: hi there
|
[21:15] jond
|
mato: i've been looking at the select.cpp and i think there is an issue
|
[21:15] jond
|
because I'm fixing something similar in the dayjob
|
[21:17] mato
|
jond: what kind of issue?
|
[21:17] jond
|
as well as checking that the array doesnt get to FD_SETSIZE, it needs to check that the fd_ in add_fd is actually greater than FD_SETSIZE
|
[21:18] mato
|
why would the actual fd matter?
|
[21:18] jond
|
FD_SET, CLR etc don't work on linux above FD_SETSIZE. it's on the man page at the bottom; I had a process with 6000 fd's open and once the sockets started connecting and getting fd > 1024 bad things happen
|
[21:19] jond
|
fd_set is normally implemented as a bitvector of fixed size
|
[21:19] mato
|
ah, you're right
|
[21:19] mato
|
yeah
|
[21:19] mato
|
good point
|
[21:20] mato
|
well, the select implementation has not really been stress-tested properly
|
[21:20] mato
|
it's only there for windows benefit (and vms I guess)
|
[21:20] jond
|
we raised call with redhat and even if you recompile glibc etc the kernel has it at 1024. My problem is that the 6000 fd's are files opened early and the sockets connect later and get to high fd.
|
[21:21] jond
|
i'm changing to poll
|
[21:21] mato
|
well, if you're dealing with 000's of fds you really should not be using select at all :-)
|
[21:21] jond
|
on solaris I believe you can just redefine FD_SETSIZE
|
[21:21] mato
|
and poll is not too performant with lots of fds either, you really want an edge-triggered interface
|
[21:22] mato
|
anyhow, thanks for the tip; i'll make a note to make a patch that adds in the appropriate assertions
|
[21:23] jond
|
yep I know, but I 'inherited' this code; we only have a 600 tcp connections, but open lots of files....
|
[21:24] jond
|
mato: edge-triggered, you mean epoll?
|
[21:27] mato
|
jond: yes
|
[21:27] mato
|
jond: unfortunately epoll never got standardised so is available only on Linux
|
[21:27] mato
|
jond: *BSD has kqueue, Solaris has Event Completion
|
[21:27] jond
|
i'll take a look tommorrow; thanks for the tip;
|
[21:27] mato
|
jond: (re the Solaris implementation, see http://developers.sun.com/solaris/articles/event_completion.html)
|
[21:28] jond
|
mato: cheers
|
[21:36] bgranger
|
mato: did you see my reply about the message related deadlock you were seeing?
|
[21:38] jond
|
bgranger: i did. is there a link to that pyzmq branch
|
[21:38] bgranger
|
Sure, right here...
|
[21:38] dermoth
|
Is there any example of XREQ/XREP out there? I've used REP/REP successfully in the past and now I'd like to add devices in the middle, and I believe I need XREQ/XREP for that.
|
[21:39] bgranger
|
http://github.com/zeromq/pyzmq/pull/18
|
[21:39] bgranger
|
Here is the class
|
[21:39] bgranger
|
http://github.com/minrk/pyzmq/blob/cd0a16729ab015ce8285843e303396ab3564173d/zmq/_zmq.pyx#L210
|
[21:40] bgranger
|
We are still doing code review so a few more things might change in the API, but I think things are mostly in place.
|
[21:40] mato
|
bgranger: hi
|
[21:41] bgranger
|
hi
|
[21:41] mato
|
bgranger: yes, i saw that, am travelling most of this week (Linux Kongress in Nurnberg, Martin Sustrik is presenting)
|
[21:41] bgranger
|
Nice
|
[21:41] jond
|
bgranger: thanks, i'll take a look. I think that function that's added in py2.7 for PendingCall looks interesting and could still provide a version which doesnt need the MesssageTrcaker
|
[21:41] mato
|
bgranger: will look at it next week once I get back home
|
[21:41] bgranger
|
Great, sounds good, by then it will probably be in our master.
|
[21:41] mato
|
bgranger: can't really test it reliably while on the road.
|
[21:41] bgranger
|
OK
|
[21:54] cremes
|
dermoth: hopefully this will help a little: http://www.zeromq.org/tutorials:xreq-and-xrep
|
[21:58] dermoth
|
[bbl]
|
[23:27] tylergillies
|
whats the difference between req/rep and upstream/downstream?
|
[23:31] vanadium
|
req/rep are bidirectional/back and forth, upstream/downstream are push/pull and one of them always sends and the other always receives
|