Wednesday April 6, 2011

[Time] NameMessage
[00:00] rizenine Anyone get the zmq gem working on Ubuntu 10.10 64bit? I'm having issues. Can't find much in a search.
[09:52] mile sustrik,
[09:53] sustrik HI
[09:55] mile hi
[09:55] mile I'm stuck with erlzmq
[09:56] mile got a minute?
[09:56] sustrik sure
[09:57] mile I suspect I forgot some include or similar
[09:57] mile since I had this working on the same machine
[09:57] mile 9> c('../src/erlzmq').
[09:57] mile {ok,erlzmq}
[09:57] mile 10> c('../test/mytest').
[09:57] mile ../test/mytest.erl:16: Warning: variable 'Msg' is unused
[09:57] mile {ok,mytest}
[09:57] mile 11> mytest:test().
[09:57] mile ** exception error: undefined function erlzmq_nif:context/1
[09:57] mile in function mytest:test/0
[09:58] mile and the line in mytest is
[09:58] mile {ok, C} = erlzmq:context(0),
[09:58] mile i have tried with 1 and without arguments
[09:58] mile but always get the same
[09:59] sustrik hm, you should speak to yrashk
[09:59] mile mytest is slightly modified test file
[09:59] mile oh, sorry, I have talked to you about nginx
[09:59] mile in the similar context
[09:59] mile thanks !
[09:59] sustrik np
[10:00] sustrik i would just file bug report on erlzmq project or somesuch
[10:00] mile I guess I made some stupid mistake
[10:00] mile so i'd like to rule that out...
[10:02] ianbarber context(0) looks fishy
[10:02] sustrik it should work, however, you will be able to use only inproc transport
[10:04] ianbarber that's true
[10:04] mile (), (1) and (0) all give the same errmsg
[10:04] ianbarber oh ok,
[10:05] ianbarber i was going to suggest () :)
[10:05] mile so I assume the problem is before it even gets there
[10:06] ianbarber do the tests that come with erlzmq2 work?
[10:06] ianbarber there is a make test I believe
[10:06] mile they do compile, I believe that is the test assumption
[10:06] mile but when I run the methods manually
[10:06] mile they produce the same problem
[10:09] ianbarber what version of zmq did you build against?
[10:12] mile hm, I'd have to rebuild it to know
[10:12] mile which is not that bad idea :)
[10:42] alkemann hey. I am looking at the worker (PHP) bit of the Divide and conquer example. why is there no if recieved bit. it seems to every loop run do both (in the code, but not in actuallity)?
[10:48] ianbarber what's the file name of that one?
[10:49] alkemann
[10:49] alkemann ianbarber: does the recv() "continue;" the loop on a "missing connection" ?
[10:54] sustrik alkemann: recv() is blocking
[10:54] sustrik it doesn't end unless there's a message afaiu
[10:54] alkemann ah doah
[10:55] alkemann so while is not about endlessly listen, is about respond to an infinite amount of recieved connections
[10:56] sustrik processing arbitrary number of messages
[10:56] alkemann makes sense now thanks
[11:01] alkemann ianbarber: when I took your advice, when I watched your talk, I had no projects for 0mq, but I've managed to get it into a project :)
[11:02] alkemann well, advice you quoted anyway :)
[11:17] mile I have tried rebuilding erlzmq2, which fetches zeromq from the git
[11:17] mile but
[11:17] mile deps/zeromq2/include/zmq.h:217: note: expected ‘size_t *’ but argument is of type ‘size_t *’
[11:17] mile ERROR: $CC -c $CFLAGS $DRV_CFLAGS c_src/erlzmq_nif.c -o c_src/erlzmq_nif.o failed with error: 1
[11:17] mile make: *** [compile] Error 1
[11:18] mile I've tried with 2.1.4. and 2.1.2
[11:18] mile both give the same error
[11:18] mile $ gcc --version
[11:18] mile gcc (Ubuntu/Linaro 4.4.4-14ubuntu5) 4.4.5
[11:18] mile anyone seen something like that?
[11:20] sustrik nope
[11:20] sustrik size_t type must be re-defined or something
[11:21] so_solid_moo mile_: I saw something like that just yesterday ;)
[11:21] so_solid_moo turned out to be a gcc bug and/or something very odd in C I don't understand
[11:21] sustrik have you solved it?
[11:22] so_solid_moo well, I had an g_array of zmq_pollitem_t, and I got that error when I had to make three references to the data
[11:22] so_solid_moo e.g. object->array->data
[11:22] so_solid_moo if I took a reference to object->array, and then used array->data, the error/warning went away
[11:23] so_solid_moo I assumed it was some kind of internal type referencing / aliasing thing in gcc getting confused
[11:24] mile I'm not sure whether the error lies in the erlang binding or in the zmq code itself...
[11:24] so_solid_moo mile_: if it's the same type of thing as mine, neither (imho)
[11:24] mile should I just get an older gcc?
[11:26] so_solid_moo maybe; I wasn't sure exactly why it emitted that error :(
[11:26] so_solid_moo the only suggestion I got was to try clang and see if that had a different opinion
[11:27] mile I'm a total c noob...
[11:27] mile so I'm kind of stuck there
[11:28] sustrik what's the compilation unit that causes the error?
[11:28] sustrik something from erlzmq?
[11:28] mile Compiling c_src/erlzmq_nif.c
[11:28] mile In file included from c_src/erlzmq_nif.c:26:
[11:28] mile c_src/vector.h:44: error: conflicting types for ‘size_t’
[11:28] mile /usr/lib/gcc/i686-linux-gnu/4.4.5/include/stddef.h:211: note: previous declaration of ‘size_t’ was here
[11:28] mile c_src/erlzmq_nif.c: In function ‘erlzmq_nif_getsockopt’:
[11:28] mile etc
[11:29] mile I guess its c_src/erlzmq_nif.c:26
[11:29] so_solid_moo ah, you have conflicting types
[11:29] so_solid_moo that's different to me I think
[11:30] so_solid_moo c_src/vector.h:44 is the error I would look at
[11:30] mile do you know how to clone an older version from the GIT?
[11:31] sustrik git checkout <tag>
[11:31] mile tx
[11:34] mile hm, no tags in erlzmq2... I guess I'll have to wait for yrashk
[11:35] sustrik ask him to make tags then :)
[11:37] mile will do :)
[13:45] pdhborges Hi guys, I'd like to ask if it's common to send the same message many times
[13:54] sustrik pdhborges: the message is nullified when you send it
[13:54] sustrik thus you have to do a copy of it before sending it twice
[13:54] sustrik use zmq_msg_copy()
[13:55] pdhborges sustrik: even if I init the message with my own buffer?
[13:55] sustrik yes
[13:55] pdhborges a provide a special free function
[13:55] pdhborges s/a/and
[13:55] sustrik zmq_msg_copy() just increments refcount on your buffer
[13:56] sustrik so that you dealloc function is called only after it's pushed to all destinations
[13:56] pdhborges what if the same message is sent from multiple sockets?
[13:56] sustrik from?
[13:57] pdhborges eg: I create a message
[13:57] sustrik you mean _to_
[13:57] sustrik the same thing
[13:57] pdhborges the send the message from 2 diferent sockets
[13:57] pdhborges a ok
[13:57] sustrik send (s1, msg);
[13:57] sustrik send (s2, msg);
[13:58] sustrik sorry, actually, it should look like this
[13:58] sustrik zmq_msg_copy (msg_tmp, msg);
[13:58] sustrik zmq_send (s1, msg);
[13:58] sustrik zmq_send (s2. msg_tmp);
[13:59] pdhborges so in that case zmq_free_fn would onl be called once
[13:59] sustrik yes
[13:59] sustrik it uses the same buffer
[13:59] pdhborges a great
[14:01] pdhborges thanks for the help
[14:01] pdhborges but is this a common use case?
[14:02] pdhborges I'm writting some bindings and I would like to know if implementing messages with shared buffers is relevant
[14:06] joelr good day
[14:07] joelr in the case of multiple clients connected to a push/pull socket, does the data get sent (pushed) when the client does a receive? Alternatively, is data accumulated on the client regardless of whether the client does a receive?
[14:09] joelr in other words, where is data accumulated with a push/pull socket before clients do a receive?
[14:18] sustrik pdhborges: it's an advanced functionality that is supported from almost no bindings
[14:18] sustrik i wouldn't care about it atm
[14:18] pdhborges sustrik: that is the only problem left for the ocaml binding
[14:20] joelr in the case of multiple clients connected to a push/pull socket, does the data get sent (pushed) when the client does a receive? Alternatively, is data accumulated on the client regardless of whether the client does a receive? in other words, where is data accumulated with a push/pull socket before clients do a receive?
[14:27] joelr pdhborges: what's the topic of discussion?
[14:28] pdhborges joelr remember the discussion about the zero copy api for ocaml?
[14:28] pdhborges well zero copy is not possible
[14:28] joelr yes
[14:28] pdhborges but copy once with reuse is certainly possible
[14:28] joelr pdhborges: why not? you can allocate the data on the c side
[14:29] pdhborges joelr yes but I have to copy the data to that buffer
[14:29] pdhborges hence it's not really zero copy
[14:29] pdhborges the python implementation
[14:29] pdhborges reuses the buffer alloc'ed by python
[14:30] pdhborges btw how is polling working for you?
[14:30] joelr pdhborges: you would write the data into the buffer from ocaml and then it won't be copied
[14:30] pdhborges yes, that is the copy once solution I was talking about
[14:31] joelr pdhborges: then zero-copy is possible
[14:31] pdhborges zero copy would be to use the result form String_val as the message buffer
[14:31] pdhborges which of course can't be done
[14:32] joelr pdhborges: what if you allocate the string on the c size and use its buffer to return an ocaml string?
[14:33] joelr on the c side
[14:33] pdhborges the message still needs to be copied to that particular buffer anyway you look at it
[14:34] joelr pdhborges: yes but only once. and you don't count that because that's not counted in zero-copy. the assumption is that you have a buffer pre-filled with data and zero-copy refers to copies made in the process of sending the buffer out through the socket
[14:35] pdhborges in that case your definition of zero copy is equal to my definition of copy once and there is no problem implementing that
[14:37] pdhborges at least as long as zmq_msg_copy is implemented with refenrece counting
[14:46] joelr pieterh: any suggestions on my question?
[14:46] joelr sustrik__: ^
[14:47] pdhborges joelr: is polling working fiine?
[14:48] joelr pdhborges: will test later this week, thanks
[15:31] Florent Hello everyone
[15:32] Florent Does anybody know if 0MQ has built-in bandwith control features?
[15:35] guido_g for pgm yes, for tcp no
[15:38] Florent Thank you. I don't know what pgm is. Let me ask my friend google...
[15:39] guido_g Florent:
[15:40] joelr help help help
[15:40] Florent thx guido_g
[15:41] Florent unfortunately I'm using the PUSH/PULL model thus PGM won't help me that much.
[15:43] Florent I'm saying this because I just read "The pgm and epgm transports can only be used with the ZMQ_PUB and ZMQ_SUB socket types."
[15:44] guido_g <Florent> Does anybody know if 0MQ has built-in bandwith control features? <- your question
[15:48] Florent guido_g: I know and I thank you for your anwser which was right.
[15:48] Florent I have to go. Goodbye and thank you again.
[18:00] zedas hey can someone point me at a sample in zmq 2.1.x that gets the raw socket for a zmq socket so that I can put it in my own epoll?
[18:01] guido_g there is no such thing
[18:01] guido_g but you can get a fd specifically for that w/ getsockopt
[18:03] zedas guido_g: right so you getsockopt with ZMQ_FD to get the FD
[18:04] guido_g right
[18:04] zedas and then ZMQ_EVENTS to get the events available.
[18:04] guido_g also right
[18:05] guido_g there is no other way because one ømq socket can hold a lot of os sockets
[18:05] guido_g at least this is what i understood
[18:05] zedas ok so what's missing is an example of: 1. get fd, 2. put in epoll/kqueue, 3. fd goes off, 4. tell zmq to deal with it.
[18:06] guido_g 4. is not needed
[18:06] guido_g all ømq io activity is done by the io threads
[18:07] guido_g and there are some experimental integrations into other event loops afair
[18:07] zedas ok, so then #4 is really just do a zmq_recv/zmq_send depending on ZMQ_EVENTS results
[18:07] guido_g exactly
[18:08] zedas ok i'm going to write an example of this with epoll
[18:09] guido_g there are some projects doing that afair
[18:09] joelr in the case of multiple clients connected to a push/pull socket, does the data get sent (pushed) when the client does a receive? Alternatively, is data accumulated on the client regardless of whether the client does a receive? in other words, where is data accumulated with a push/pull socket before clients do a receive?
[18:09] guido_g have a look at the mailing list
[18:12] guido_g joelr: the data is sent to a _connected_ peer as soon as possible, if the underlying send-buffers are full, queueing takes place in the sender; the reciever fetches the data and puts it in his queue
[18:13] sustrik zedas: have a look at src/zmq.cpp
[18:13] joelr guido_g: thanks, that's what i thought. and the send buffer will be full if receivers have not received, right?
[18:13] sustrik there's an implementation of zmq_poll using ZMQ_FD and ZMQ_EVENTS
[18:13] guido_g joelr: no
[18:13] sustrik one important thing is to take care the the FD is edge-triggered
[18:13] guido_g joelr: receive is asynchronous and done in the ømq io thread
[18:14] joelr guido_g: so when i call recv in the client, it's just a data transfer into the calling thread?
[18:15] joelr guido_g: what i'm trying to get at is whether data will be sent round-robin to the zmq thread (background) of the the connected client, regardless of whether the client calls zmq_recv
[18:16] joelr will it be sent or will it not be sent?
[18:16] zedas sustrik__: i am, but that 5 nested loop poll isn't a very good example (or a very good poll to be honest)
[18:16] sustrik i don't know whether there are any simple examples
[18:16] sustrik but the logic is simple
[18:17] sustrik 1. check ZMQ_EVENTS to find out whether you can read/write immediately
[18:17] sustrik 2. if not, poll on ZMQ_FD
[18:17] sustrik 3. goto 1
[18:17] joelr guido_g: what do you say?
[18:19] zedas sustrik__: yep i'll write a clear simple example that doesn't wrap a two for-loops in an error stuffing while loop inside two #defines that should be functions.
[18:21] joelr sustrik__: with a connected pull socket, will any data be pushed to the client if they do not call zmq_recv?
[18:21] sustrik zedas: thanks
[18:21] sustrik two important points are
[18:21] sustrik 1. you have to check EVENTS first, otherwise you can deadlock
[18:22] sustrik 2. if FD is signaled, it doesn't necessarily mean, there a message available
[18:22] sustrik you still have to check EVENTS
[18:22] sustrik joelr: yes
[18:23] joelr sustrik__: so in the case of a durable socket, will this data be resent by the sender if the receiver crashes before calling zmq_recv?
[18:24] sustrik no, it won't
[18:24] sustrik there's an ticket for that
[18:24] sustrik but it's not coded
[18:25] joelr sustrik__: oh, ok then. because it would seem that there's not much of a point in a durable socket if that happens
[18:25] guido_g sorry, life person visit here
[18:25] joelr sustrik__: thanks for the info
[18:26] sustrik yes, there's not much point in durable sockets
[18:26] sustrik but they can't be removed because of backward compatibility
[18:26] joelr sustrik__: what is the suggested workaround until this is fixed? push/pull sockets are awfully nice because of load balancing
[18:27] joelr well, i guess that's the wrong question
[18:27] joelr hmm
[18:27] joelr sustrik__: some kind of a "work received, work completed" feedback system to the sender?
[18:29] sustrik to be frank, full reliability is impossible
[18:29] sustrik even if there's resend functionality
[18:29] sustrik the message that was being processed by consumer at the point of crash
[18:29] sustrik will be lost
[18:29] joelr sustrik__: so a feedback system then for each item of work completed?
[18:30] joelr sustrik__: with a resend of some kind through a direct (REQ?) channel?
[18:30] sustrik depends on your topology
[18:30] sustrik if there's single origin of all messages, they can be numbered
[18:31] sustrik and if there's single final destination for messages, the missing ones can be detected
[18:31] sustrik and the source can be asked to resend them
[18:31] joelr sustrik__: i'm trying to design the topology. it certainly starts with mongrel and from the handler of the mongrel messages there are several other services to be accessed
[18:31] joelr i'm thinking that all end-services should just be rep since they do a thing and send results back
[18:32] sustrik the question is: what kind of reliability do you need?
[18:32] sustrik when are you willing to accept a message loss?
[18:32] joelr sustrik__: i'm not. a web request needs to be serviced so a resend is needed.
[18:32] joelr assuming all services come at least in pairs
[18:32] joelr and you can obtain the endpoints for each service via a name service
[18:33] joelr so a message loss is not acceptable
[18:34] sustrik impossible, sorry
[18:34] sustrik no system is 100% reliable
[18:35] joelr 99.9% would be ok
[18:35] joelr or something like erlang's 99.999%
[18:35] sustrik yes, that's acceptable
[18:35] sustrik so, you have to make your app bug free
[18:35] sustrik buy an UPC unit
[18:36] zedas joelr: what you want to get that is set the recv_ident and send_ident on both sides, in the handler and in mongrel2, then it's at that level
[18:36] joelr sustrik__: you are pontificating whereas i'm looking for practical advice with topology ;-)
[18:36] sustrik that would do for 99.9
[18:36] sustrik the point is you have to have reasonable expectation
[18:36] joelr sustrik__: yes
[18:37] sustrik so back to the original question
[18:37] sustrik is there a single source of all messages>
[18:37] sustrik ?
[18:37] zedas joelr: another thing is, if the message is of the kind where it *must* be delivered, then you'll want to send it to something that stores it. then other things are notified it's stored and work on it.
[18:37] joelr sustrik__: mongrel, i suppose
[18:37] joelr zedas: i'm processing mongrel requests
[18:38] joelr but the handler needs to call other services to assemble a reply
[18:38] zedas joelr: yep, i know. so let's say there's a particular handler that must always process everything it gets. that handler should store it, then work on it and it should be setup so that it's async.
[18:38] joelr zedas: why async?
[18:39] zedas joelr: async works better with http and with complex/long running tasks.
[18:39] zedas you receive the request and then set it up and get out of there, notifying the front-end immediately.
[18:39] joelr zedas: i suppose because the handler can continue dispatching work and can always route it back to the right mongrel via the identity
[18:40] zedas now, you are free to work on it as you need, scale it out, carve it up, and the end result is notifying the user, via the GUI, that it's done.
[18:40] zedas bingo. the key is to switch from "this message must always go trhough like the emails!" to "after I store this, a bunch of people work on it, and I'll tell you when it's ready"
[18:40] joelr zedas: well, in my case i cannot notify the front-end (browser) with json until i complete processing the request
[18:40] joelr sustrik__: mongrel is the single source
[18:41] zedas that's fine, in mongrel2 the socket stays open until you close it, and the receiver doesn't need to be the reply entity
[18:41] zedas so, have one handler receive and store it right away. then hav eother handlers work on it. then have the final master send the response.
[18:41] joelr zedas: if the receiver does not need to be the reply entity, then something has to know the mongrel endpoint or be connected to it, right? the sub end of it
[18:42] zedas however, that's going to fail. any time you have these kinds of requests, if the GUI isn't designed to assume a later async response, then you'll have problems.
[18:42] joelr i don't see how you can do it another way
[18:42] zedas you totally know the endpoint, that's why you have the uuid, conn_id, and access to the sqlite database.
[18:42] zedas do a query on it, pull it up and then tlak to it. I do this in Tir all the time.
[18:42] joelr zedas: if mongrel just sits there waiting for a reply on its pub socket then i can always send the reply somewhat later, no?
[18:43] joelr after i assemble it
[18:43] zedas totally. mongrel2 has no notion of "request/reply". you can have 10 other handlers reply
[18:43] zedas another batch of advice: you're over optimizing right now. you should just get the thing working with one handler, *then* figure out how to make it never fail.
[18:43] joelr my question is really that of how to scale the various services on the backend. it seems that having several endpoints and instances for each services should do it (freelance pattern)
[18:44] zedas the reason is it's *way* easier for me to tell you how to fix something you ahve that's simple but broken, but we could talk for days about hypothetical failure modes that might or might not happen without code.
[18:44] joelr zedas: i'm acquiring knowledge how to do the never fail but i won't be implementing it for now
[18:44] joelr zedas: scalability is what i'm interested in
[18:44] joelr scaling by adding more instances of each of my services
[18:44] joelr this is an ad network, btw
[18:45] zedas joelr: alright and i'm saying: go code it. it's possible, and you could code all the scenarios you want in about a day.
[18:45] zedas that's why zeromq works so well. you can prototype any bizarre network scalability mess you want.
[18:45] joelr so i have services like stats that keep track of clicks, impressions, etc. people ask this service for click-through rate for a given page displayed
[18:45] zedas joelr: also read
[18:46] joelr how do i scale it? it seems like i could use the freelance pattern where there are several instances of the service
[18:46] joelr and simply have each service use a REP socket
[18:46] joelr valid?
[18:46] zedas joelr: you scale it with
[18:47] zedas hehe, seriously, you're at a point where you should code up a tiny prototype to see what's possible.
[18:47] sustrik zedas: btw, i am having a talk about 0mq in moscow, but it's a webdev conference, so i would like to say something about mongrel2
[18:47] sustrik what should i read about it to get a picture?
[18:47] joelr zedas: i'm at a point where i have already coded my tiny prototypes and i'm looking at the next step
[18:48] zedas sustrik__: oooh moscow, always wanted to go there. what you looking to talk about re:mongrel2? how zmq is used?
[18:48] joelr and so i'm looking know-how
[18:48] sustrik yes, more or less
[18:48] zedas sustrik__: you seen my presentation for pycon? i can shoot you the slides.
[18:48] sustrik yep, seen it
[18:48] sustrik but maybe some more technical detail
[18:49] sustrik should i read the docs?
[18:51] zedas sustrik__: docs would work, but there's plenty of code samples in various languages. i'd take a look at and and as examples of how it's being used
[18:51] sustrik ok, will do
[18:51] sustrik thx for advice
[18:51] zedas sustrik__: also how zeromq simplifies a common pain in webdev: background tasks:
[18:52] zedas mongrel2 by itself might not be attractive, but the pattern of doing an async background task of zeromq is killer for web dev.
[18:52] zedas as joelr was just fretting over. it's pretty much *the* pain in the web's ass.
[18:52] sustrik aha, that looks like something that may be interesting for the audience
[18:53] zedas sustrik__: in Tir it's similar and read Builtin Background Tasks
[18:53] sustrik ok, I'll do some reading, I will ask you if i am not sure of some aspects of thing
[18:54] zedas go for it.
[18:55] zedas joelr: feel free to continue this in #mongrel2, seems like more of a topic for there.
[18:55] joelr zedas: fair
[18:55] sustrik joelr: sorry
[18:55] joelr sustrik__: no problem
[18:56] sustrik zedas is more relevant for this topic than i am :)
[19:56] achin if i've got a question about building pyzmq, can i ask in here?
[19:56] sustrik sure
[19:59] achin i build libzmq from the latest source, and then i ran the following issue:
[20:00] achin i think some cython magic has gone awry, but i don't have any experience with cython and all the fancy stuff it does
[20:08] sustrik ah
[20:08] sustrik the master is now undergoing changes
[20:08] sustrik use 2.1.4 instead
[20:10] achin same thing i'm afraid (using the 2.1.4 branch of pyzmq)
[20:29] sustrik same errors?
[20:30] sustrik you must have headers from master installed somewhere
[20:30] sustrik the errors in the pastebin are complaining about the changes done in master
[20:35] achin i did a clean, and another configure:
[20:36] sustrik it looks like the pyzmq is getting the header file from master still
[20:36] sustrik check for zmq.h files on your disk
[20:38] achin maybe i'm just confused. i'm still using master from libzmq -- is that appropriate? or should i use something older?
[21:54] freeformz Has anyone used rzmq with sinatra in any way successfully? Including testing? I'd love to understand your usage pattern