IRC Log

Tuesday January 4, 2011

[Time] NameMessage
[05:42] jugg I've done some work updating the zmq erlang bindings, for the most part things are working well now, but I'm running into issues with events on the socket FD (that was retrieved using ZMQ_FD getsockopt). question: Can the FD be used in poll/select in one thread while the Socket is being used in other thread? eg. the erlang vm sits on the FD in the poll/select call while waiting for an event and in the mean time other socket operations may be performe
[06:45] sustrik jugg: no
[06:45] sustrik sockets are not thread safe
[06:46] sustrik you can access a socket from at most one thread at a time
[06:48] jugg Is this zmq specific, or (I've not yet been able to find an answer) is it the case that a normal file descriptor would have problems if one thread used it in a poll() call, while another thread tried to read/write to it?
[06:49] sustrik normal FDs are thread safe
[06:50] sustrik 0mq is special as it uses lock-free algorithms
[06:50] sustrik thus, it cannot synchronise the access using mutex or somesuch
[06:51] sustrik you can apply a mutex on top of it yourself, if that's what you need
[06:51] sustrik but be aware of the performance impact
[06:51] jugg What makes the socket FD different than a normal FD in this regards?
[06:51] jugg No, that isn't an option for the erlang bindings...
[06:59] neopallium jugg: the 0mq FD is just a socketpair to sent notices that there are messages/commands to read from the socket's queue.
[07:00] neopallium the message data you get from recv() is transferred in-memory from a lock-free queue, not over the socketpair.
[07:02] neopallium jugg: are you making a erlang port or a linked-in driver?
[07:03] jugg linked-in driver - its based off of the broken erlzmq bindings.
[07:10] jugg So if the socket FD is a socketpair, what makes it not usable in a poll call from one thread while another thread uses the socket? Does zmq listen for events on the socket internally (outside of zmq_poll itself)?
[07:11] sustrik jugg: yes, it does so in zmq_send, zmq_recv and alike
[07:12] jugg ok... well, time to approach the erlang binding architecture differently then... :/
[07:14] neopallium sustrik: if the polling thread just calls poll() and not any zmq_* functions like zmq_poll(), would it still cause an issue?
[07:14] jugg if other parts of zmq consume the socketpair events, then yes...
[07:14] neopallium jugg: what is your polling thread doing with the events it gets back from the 0mq FD?
[07:15] jugg Notifies the socket owner thread and event occurred.
[07:16] jugg and => an
[07:16] neopallium ok, even if the owner thread gets a old POLLIN or POLLOUT event it can still try to call zmq_send/zmq_recv() with the NOBLOCK flag.
[07:16] neopallium it will just get an EAGAIN if it had already comsumed the event.
[07:17] jugg the problem is that it does not get a POLLIN/OUT event - as it appears the zmq internals are consuming those events...
[07:17] jugg which from sustrik's statement confirms this.
[07:17] neopallium do you use the NOBLOCK flag?
[07:18] jugg I don't see the relevance... the problem is the erlang poll is not being signaled an event occurred...
[07:18] neopallium your owner thread should be calling zmq_send()/zmq_recv() with the NOBLOCK flag, and only if it get EAGAIN, do it wait for an event notice from the polling thread.
[07:18] sustrik neopallium: getsockopt(ZMQ_EVENTS) consumes events from the socket pair
[07:19] neopallium sustrik: yes, and he should only call that from the owner thread not the poller thread.
[07:20] sustrik jugg: why do you need to combine recv and poll from different threads
[07:20] sustrik (i know, erlang is tricky)
[07:21] sustrik neopallium: i assume the problem is how erlang VM works
[07:21] sustrik it's kind of thread-agnostic
[07:22] neopallium jugg: have you tried an external C port? it would make this must similar.
[07:23] neopallium sustrik: yeah, the book I have on erlang doesn't have much details about how the linked-in driver interface is effected by any internal threads that the erlang VM might have.
[07:25] sustrik the erlang binding is a troublesome one
[07:25] sustrik originally it was written by shammika pathirana
[07:25] jugg the only way to wait on an event in erlang is to provide the fd to the erlang vm. The erlang vm, then calls the driver callback routine input/output_ready depending on the event type. The callback is an arbitrary thread. That callback checks the fd if an even actually occurred on the socket using getsockopt(ZMQ_EVENTS), and if so, then sends a notification back into the erlang world notifying an erlang process which owns the socket. That erlang pr
[07:25] sustrik who had no erlang experience at the time afaik
[07:26] sustrik then it was fixed by serge aleynikov
[07:26] sustrik later on we've created some fixes when working on RabbitMQ/0MQ bridge with VMWare
[07:26] neopallium jugg: are you sure the vm is using a different polling thread from the callbacks?
[07:26] sustrik this are not yet in
[07:26] sustrik etc.
[07:27] jugg neopallium, the callback gets called from arbitrary threads... I don't know for sure if the poll is happening in the same thread or not.
[07:28] sustrik what about an alternative architecture: poll on FD, when signaled remove it from the pollset then read, when read is done, return it to the pollset
[07:29] neopallium jugg: why not do the zmq_recv() in the output_ready() callback instead of sending a notice to the erlang process (which is not a real thread/process).
[07:30] jugg sustrik, I'm already half way down that path...
[07:30] neopallium even if you are sending a notice back to the erlang process (saying hay you can read now), that erlang process will just send a port command which will be delivered to the port
[07:30] jugg neopallium, that has issues with multi-part messages...
[07:31] neopallium the ports callbacks should be restricted to one thread.
[07:31] neopallium I don't see any warnings that linked-in drivers need to be thread-safe.
[07:32] neopallium jugg: do you deliver each part of the message in a different message back to the controlling process?
[07:33] jugg each erlzmq recv call acts just like zmq_recv. It'll return each part separately.
[07:34] sustrik jugg: btw, there's a problem with erlzmq project now; as i've mentioned dhammika is not really an erlang guy and serge has unrelated problems now that prevent him from doing stuff
[07:35] sustrik in case you are interested i can give you commit access...
[07:36] jugg sustrik, let me get this working successfully, and I'll ping you then for access.
[07:37] sustrik jugg: ok, good
[07:38] sustrik there's also VMware patch that solves the sender-side backpressure IIRC
[07:38] jugg sustrik, just to clarify something.. you said getsockopt(ZMQ_EVENTS) consumes the socketpair events... do you mean payload or events that would trigger a waiting poll call?
[07:39] sustrik it consumes internal 0mq events from the socket pair
[07:39] neopallium jugg: it reads from the socketpair which will consume the POLLIN event from the FD
[07:40] sustrik let me give an overvirew of the achitecture
[07:40] sustrik the actual TCP sockets are handled by background threads
[07:40] sustrik making sending/receiving messages asynchronous
[07:41] sustrik the user's thread communicates with background (I/O) threads using so called 'commands'
[07:41] sustrik these are sent via a socketpair connecting the user thread and the I/O thread
[07:42] sustrik example of a command sent from I/O thread to user's thread would be "there are messages available for reading"
[07:43] sustrik when user's thread get that command it knows it can unblock (in case it's blocked in recv) and read new messages from the lock-free queue
[07:43] sustrik that's more or less it
[07:45] jugg are socketpairs ever shared with multiple zmq sockets?
[07:46] jugg or does every zmq socket have its own socketpair?
[07:46] sustrik every socket has it's own socketpair
[07:47] sustrik this is needed to be able to migrate sockets between threads
[07:47] sustrik so, to be precise, in 2.0.x multiple sockets share single socketpair
[07:47] sustrik in 2.1 there's one socketpair per 0mq socket
[07:50] jugg ok. Thanks, this helps my mental picture. I have a couple of approaches to test out now.
[07:51] sustrik good luck :)
[07:52] sustrik ah, btw, erlang binding works with 2.1 only anyway
[07:52] sustrik so you don't have to take the case where the socketpair is shared between sockets into account
[07:54] jugg yes
[07:59] neopallium jugg: I think you should only register events with driver_select() when the 0mq socket is blocked as reported by ZMQ_EVENTS.
[07:59] neopallium and when you are waiting for an event back from the vm don't call zmq_recv() or ZMQ_EVENTS until the event comes back.
[08:00] jugg that is interesting
[08:00] neopallium that way the owner thread can't consume events from the socketpair.
[08:01] neopallium I found the details about driver's and erlang threads.
[08:01] neopallium and by default the driver will only have one thread access it at a time.
[08:02] neopallium you can do locking at driver or port level, the default is locking at the driver level.
[08:03] neopallium if you don't share 0mq sockets between ports it should be possible to use port-level locking.
[08:04] neopallium also remember to remove the event notice from the socket after receiving the event and mark the socket as readable/writable.
[08:05] neopallium sustrik basically said the same thing earlier about removing the socket from the pollset (i.e. from driver_select()) when it is readable.
[08:06] jugg I'm not sure what you meant by "remove the event notice"... are you talking an implementation concept, or an erlang requirement?
[08:07] neopallium jugg: driver_select(port, event, mode, /* put 0 here. */)
[08:08] neopallium that last arg. is for clearing the event (i.e. removing it from the pollset)
[08:09] jugg yes. But my understanding is (although it has been hard to find supporting evidence) that it isn't a requirement normally to do that in order for the erlang vm to report subsequent events on a fd.
[08:10] neopallium jugg: is your current problem that you are not receiving an event from the vm?
[08:11] jugg yes, but I think between sustrik's comment on getsockopt(ZMQ_EVENTS) consuming events, and his description on how things work, I can see why that is the case...
[08:11] neopallium 0mq sockets are edge-triggered, that means you need to read from them until they will block (i.e. you get EAGAIN)
[08:11] jugg yes
[08:12] neopallium when your driver call getsockopt(ZMQ_EVENTS) and it says there is a message to read, then you need to mark the socket as readable, until you read a full message.
[08:13] neopallium once you read a full message call ZMQ_EVENTS again to check if there is another message to read, if it reports "no messages", then you wait for a read event from the vm.
[08:14] neopallium I think there was an example in the guide that use ZMQ_EVENTS like that.
[08:14] sustrik yes, more or less
[08:15] sustrik but you don't have to store the "readable" flag yourself
[08:15] sustrik you can retrieve it from 0mq by calling getsockop(ZMQ_EVENTS) actually
[08:15] jugg yes, I think the key is to implement a poll method which checks ZMQ_EVENTS prior to passing the fd to the erlang vm, and only do so if nothing is pending.
[08:15] sustrik yes
[08:17] neopallium jugg: is this driver pushing messages out to the controlling erlang process? or does the erlang process block in a recv() type function for each message?
[08:22] jugg the driver returns messages only if the erlang process requests it via recv(). Of course, driver does not block in the recv() call, as that'd freeze the vm. It later pushes the incoming message when it comes available - thus the driver must use the fd to be notified of such an event. A gen_server is used to ensure further requests to the driver is not made, and the calling erlang process is blocked in the gen_server call until the driver eventually
[08:22] jugg of course the erlang process can call a non blocking recv as well.
[08:23] jugg (the old erlzmq did not support that last)
[08:27] jugg sustrik, does one only use the socketpair fd with POLLIN? even if waiting for the zmq socket to become writeable?
[08:28] neopallium jugg: so the erlang sends the port a {call, recv} type message telling it to call zmq_recv() and send back a part of the message.
[08:28] sustrik jugg: yes
[08:28] sustrik POLLOUT makes no sense
[08:28] jugg sustrik, ok
[08:29] jugg neopallium, the driver will always call zmq_recv with NOBLOCK. But if the erlang requests did not specify NOBLOCK, and zmq_recv returns EAGAIN, then the driver calls driver_select on the socket fd, and waits for an event, at which point it reads from the socket and returns the received message part to the erlang process.
[08:29] neopallium sustrik: so if the 0mq socket blocks on zmq_send(), do you have to wait for a POLLIN from the socketpair and the check with ZMQ_EVENTS if the socket is writable?
[08:30] sustrik yes
[08:30] sustrik the underlying socketpair is actually used for uni-directional transfer of commands
[08:30] neopallium ok, I need to fix my Lua async handler for 0mq sockets.
[08:30] sustrik so POLLOUT is always signaled
[08:31] sustrik i think it's mentioned in the reference
[08:31] sustrik let me check
[08:31] jugg sustrik, the docs could be clarified on that regards... It kind of says so, but it was unclear.
[08:31] neopallium yeah, I am basically doing a POLLOUT on the socketpair when zmq_send() returns EAGAIN.
[08:32] neopallium it is hard to test blocking on zmq_send()
[08:32] jugg indeed :/
[08:32] sustrik "0MQ library shall signal any pending events on the socket in an edge-triggered fashion by making the file descriptor become ready for reading"
[08:33] sustrik how would you reprharse that?
[08:33] neopallium .... signal any pending events from getsockopt(ZMQ_EVENTS...) when the FD is readable?
[08:34] sustrik i don't understand that :)
[08:34] sustrik maybe 2 sentences?
[08:34] neopallium it needs to say something about calling getsockopt(ZMQ_EVENTS...) for the ready events.
[08:35] sustrik "The ability to read from the returned file descriptor does not necessarily indicate that messages are available to be read from, or can be written to, the underlying socket; applications
[08:35] sustrik must retrieve the actual event state with a subsequent retrieval of the ZMQ_EVENTS option."
[08:35] sustrik see zmq_getsockopt(3)
[08:36] neopallium how about something like: "0MQ library shall signal any pending events by making the FD readable, getsockopt(ZMQ_EVENTS...) returns which events are pending."
[08:37] sustrik that's better
[08:37] neopallium yeah, but not perfect
[08:37] sustrik but kind of duplicates what the note above says
[08:38] sustrik have a look at whole ZMQ_FD section and feel free to propose a better wording
[08:40] neopallium hmm, yeah I guess the original dues say that in the Note
[08:41] neopallium maybe just add a note that checking for POLLOUT on the FD is use-less for finding out if it is writable?
[08:42] jugg is it the case to mark the socketpair FD read, one must call getsockopt(ZMQ_EVENTS), else subsequent calls to poll will always return immediately?
[08:44] neopallium jugg: if you get a POLLIN event from poll() you will need to call getsockopt(ZMQ_EVENTS) to clear that event, or you will just keep getting the POLLIN.
[08:46] neopallium I think from a simple event loop it would be best to call getsockopt(ZMQ_EVENTS) on each POLLIN event and then dispatch the real POLLIN/POLLOUT events as returned from getsockopt(ZMQ_EVENTS).
[08:47] neopallium that is what I am going to try with my async. handlers.
[08:49] sustrik jugg: the event is cleared also when you call non-blocking recv() which returns with EAGAIN
[09:26] mikko good morning
[09:53] sustrik morning
[09:57] mikko hmm, [zeromq-dev] Segfault
[09:57] mikko looks like to be related to the HWM bug we discussed ages ago
[09:58] sustrik mikko: the one you've gave me the test case for?
[09:58] mikko sustrik: i think so
[09:58] sustrik i've looked into that but the problem was not obvious
[09:58] sustrik i'll have to look at it further
[09:58] mikko man, year after year it's harder to come back to work after holidays
[09:59] sustrik same here :)
[10:15] neopallium I hit this assertion in zeromq: Assertion failed: msg_->flags & ZMQ_MSG_MORE (rep.cpp:80)
[10:15] neopallium that is from a REP socket with a XREQ client.
[10:16] neopallium is it bad to connect a XREQ socket to a REP socket?
[10:18] neopallium hmm, looks like XREQ -> REP is a supported setup.
[10:23] jugg I believe REP socket expects at least two message parts, but three is needed. <msg id> <empty msg> <msg body>. Both <msg id> and <empty msg> should be sent with SNDMORE flag set.
[10:26] jugg basically, your application code on the xreq side passes in <msg body> (in one or more parts). The xreq wrapper layer must then provide an identifier for that <msg body> and send that along to the rep socket, along with an empty message so that the rep socket and delineate between id parts and body parts. When the rep socket responds, your xrep wrapper layer pulls off the msg id part and looks up who to route the body part to.
[10:28] jugg "...so that the rep socket can delineate.."
[10:29] neopallium jugg: yup, REQ/REP sockets hid the "address" part of the message, XREQ/XREP expose the "address" part.
[10:30] neopallium I am trying a manual example with the plain lua-zmq bindings to make sure it is not just my use of zmq that is cause it.
[10:30] jugg so, you were already sending along the message id and blank msg?
[10:30] jugg with SNDMORE set?
[10:31] neopallium testing that right now, making a minimal example.
[10:41] neopallium jugg: https://gist.github.com/764629
[10:42] neopallium also sometime I got a crash on the XREQ side.
[10:42] neopallium but not an assertion, just a core dump.
[10:43] jugg I believe the delim needs to be empty, zero bytes. You are sending a single byte.
[10:43] jugg try: s:send("", zmq.SNDMORE)
[10:44] neopallium ah, yeah, I just tried that and it worked
[10:44] jugg cool
[10:44] neopallium I don't know where I read to send "\0" as the delim. part
[11:05] orestis Hello, I'm wondering how 0mq fits in with a GUI event loop - is it safe to make 0mq requests from the GUI thread?
[11:05] mikko orestis: you mean whether it would block?
[11:06] mikko orestis: can you integrate external filehandle polling to your gui event loop?
[11:06] mikko you can get ZMQ_FD from the socket and use that event loop
[11:07] mikko then in the callback check for ZMQ_EVENTS
[11:29] orestis Thanks. That looks a bit too cutting-edge for my taste. I guess my original question was, are the 0mq send/receive calls async? I'm a bit confused because there are mentions of background threads.
[11:33] mikko orestis: send/recv is async yes but there is a possiblity that the calls block
[11:33] mikko unless ZMQ_NOBLOCK is used
[11:37] orestis Thanks. In general, is 0mq a bit more low-level for doing async I/O from something like Twisted (that gives you Deferreds, Callbacks etc)?
[11:38] mikko orestis: i guess that is true. then again as far as i understand twisted is more general purpose ae framework where as 0mq is messaging stack
[11:41] orestis yes, apparently. I mainly wanted to use 0mq to split a monolithic gui app that does heavy I/O into separate processes. I definitely want to use Twisted for Deferreds and Callbacks (and Twisted plays well with GUIs) so I was wondering how to bring everything together.
[11:42] mikko orestis: you should be able to integrate 0mq into twisted
[11:42] mikko using the methods i mentioned earlier
[11:44] orestis mikko: they're not in the stable version though, are they?
[11:45] mikko orestis: if you mean 2.0.x then no
[11:45] mikko but 2.1.x seems fairly stable
[11:45] mikko i use github trunk in all my developments
[12:14] orestis mikko: thanks for your input. I'll see what I can come up with.
[12:21] jugg The documentation for ZMQ_IDENTITY is not clear to me. I assume that the zmq context has to be maintained in order for a socket's identity to be re-usable? But terminating the zmq context will lose the identity persistence?
[15:55] Seta00 how do I build 0MQ with MinGW?
[15:55] mikko Seta00: configure && make && make install i guess
[15:55] mikko Seta00: i got mingw build running on linux machine
[15:55] Seta00 :P
[15:56] mikko haven't tried on windows in a while
[15:56] mikko i got mingw32 on linux and VS2008 on windows in daily builds
[15:56] mikko maybe mingw32 on windows would be a good addition
[15:57] Seta00 definitely :)
[16:00] Seta00 working with various libraries can get messed up really fast
[16:12] Seta00 msg_store.cpp:279: error: ISO C++ does not support `long long'
[16:12] Seta00 why aren't things easier? hehe
[16:13] sustrik Set00: is that 2.0.10?
[16:13] Seta00 sustrik, yes
[16:13] sustrik i recall the long long issue was being solved
[16:13] sustrik maybe it's fixed in 2.1
[16:14] sustrik mikko: don't you remember by chance?
[16:14] Seta00 I'll try
[16:26] Seta00 2.1.0 gives me ".../mingw/lib/dllcrt2.o: No such file or directory" when linking
[16:27] sustrik :|
[16:28] sustrik no idea what that is
[16:28] sustrik as for the long long problem
[16:29] sustrik iirc the problem is that long long is not part of C++ standard, yet it it used in Win32 signatures
[16:29] sustrik Win32 API
[16:30] Skaag I wonder if anyone else is using google protocol buffers to send messages over zmq
[16:31] Seta00 I'm trying with uint64_t
[16:33] Seta00 doesn't work
[16:34] Seta00 how can I remove -Werror?
[16:34] Seta00 other than manually editing the makefile?
[16:35] Seta00 long works fine, but msys doesn`t find that DLL
[16:40] sustrik no idea, try checking configure.in
[16:41] Seta00 forgot to configure fstab
[16:42] mikko sustrik: yes
[16:42] mikko ah, it was answered
[16:42] mikko long long problem is with 2.0.x
[16:43] mikko it's fixed in 2.1.0
[16:43] sustrik thx
[16:46] mikko Skaag: there has been a couple of mentions about that
[16:46] mikko Seta00: i can test mingw32 build when i get home
[16:47] Seta00 mikko, it worked after I configured my fstab on MSYS
[16:47] Seta00 thanks sustrik and mikko
[16:47] sustrik you are welcome
[16:47] Seta00 now I need qmake to link to it :P
[16:48] Seta00 which shall be solved in #qt
[16:48] Seta00 :)
[16:48] mikko i remember someone doing this before
[16:48] mikko with qt toolchain
[16:48] mikko that sparked the whole fixing of mingw32 builds
[17:28] s0undt3ch with pyzmq, on a publisher subscriber model, I'm setting the publisher's identity, how can a subscriber retrieve that identity
[17:28] s0undt3ch ?
[17:31] sustrik it can't
[17:33] s0undt3ch hmmm
[17:33] s0undt3ch I need to pass that on the message then
[17:34] sustrik yes, the identity is only internel per-hop mechanism
[17:34] sustrik it's not passed all the way down from publisher to subscriber
[17:39] s0undt3ch sustrik: same for req/rep right?
[17:41] Seta00 s0undt3ch, are you on windows?
[17:44] sustrik same for all patterns
[17:45] sustrik req/rep is special in that identities are stuck to the request as it passes individual nodes
[17:46] sustrik but there's still no special mechanism to inspect peers' identities
[17:48] s0undt3ch Seta00: nope
[17:49] Seta00 just checking if someone has a pyzmq build for Windows
[18:54] Joes Hello. It is very tempting for me to use ZeroMQ, but I'm trying to figure out how architecture would look like. So, we have chat server (a-la IRC) with chat rooms.
[18:54] Joes Users can join multiple rooms and I need to route messages between users in same room.
[18:54] Joes As far as I understood tutorials, I need 2 connections
[18:55] Joes one pubsub to send room events back to users
[18:55] Joes and one pubsub to send events from users to rooms?
[18:55] Joes can it be done with only one connection?
[18:55] Joes (or socket)
[19:16] mikko Joes_: you would need two
[19:17] Joes in terms of low level communication with tcp transport - two tcp ports on the server will be used
[19:17] Joes correct?
[19:17] Joes (firewall setup, etc)
[19:18] mikko Joes_: assuming you bind the two sockets then yet
[19:18] mikko yes*
[19:18] Joes ok
[19:18] Joes and second question - any bindings for Javascript?
[19:18] mikko i think someone was working for node.js bindings
[19:18] Joes using WebSockets/socketjs/etc
[19:18] Joes saw them
[19:18] Joes includes native C code
[19:19] Joes is wire protocol documented?
[19:19] mikko yes, i think it is
[19:19] mikko gimme a sec
[19:20] mikko here is the message format http://rfc.zeromq.org/spec:2
[19:20] Joes thanks
[19:20] mikko let me see if the rest is documented
[19:21] Joes going to see how long it will take to write dumb client in JS
[19:21] Joes there's one for AMQP, but nothing for ZeroMQ
[19:21] Joes and I don't need 95% of AMQP features...
[19:22] mikko are you going to do communications from browser straight to 0mq?
[19:22] Joes well, that would be ideal
[19:22] mikko i don't think that's going to work
[19:23] mikko at least with websockets
[19:23] mikko as websockets have their own framing as well
[19:23] Joes instead of making own wire protocol
[19:23] Joes hm
[19:23] mikko have you looked into mongrel2?
[19:24] Joes I did
[19:24] Joes however, I can't find any benchmarks
[19:24] Joes or who's using it in production
[19:24] mikko it's very new tech, i would be surprised if there are large installations yet
[19:25] mikko and i tend to find benchmarks mainly useless as there are a lot of people who don't know how to benchmark the software
[19:25] mikko or optimize it for benchmarking
[19:25] Joes well, at least some rough numbers would help
[19:25] Joes like faster than X Y times
[19:27] mikko but very often i find they are comparing apples to oranges
[19:27] mikko one tool with highly optimized config against incorrect config of other
[19:27] Joes I see.
[19:28] mikko it's very hard to create good benchmarks
[19:32] mikko but communicating from browser you most likely need some sort of middle piece
[19:32] mikko probably doesn't need to be complicated but something that converts websocket messages to 0mq messages
[19:32] Joes Let's assume I have middlepiece - some kind of broker between browser and 0MQ
[19:32] Joes (thought about it)
[19:32] Joes with some simple wire protocol
[19:32] sustrik i think that's mostly what mongrel2 does
[19:33] Joes but I have following issues:
[19:33] Joes 1. Who will guarantee delivery, if client drops connection
[19:33] Joes (I will need to queue up messages for this client till his next connection)
[19:33] Joes well, there some other issues, but I don't remember them all :-)
[19:33] mikko if client drops connection do you want to queue messages?
[19:34] mikko for how long time after client leaves?
[19:34] Joes I don't know if client dropped or temporary lost connection to the server and will reconnect shortly
[19:34] sustrik can be done using ZMQ_IDENTITY
[19:34] Joes em
[19:34] Joes let's say I'm am the server which handles requests from browsers.
[19:34] Joes I accepted raw TCP connection (or websocket connection)
[19:35] Joes so, for this connection, I will create 0MQ socket (or two, but it does not matter now)
[19:35] Joes and will connect to the chat server
[19:35] sustrik right
[19:35] Joes if incoming (browser) connection will drop
[19:35] Joes I should close socket?
[19:35] Joes (0MQ socket)
[19:36] sustrik no, 0MQ socket represents all the connections
[19:36] sustrik so closing it means closing all the connections
[19:36] Joes hm
[19:36] Joes 1 socket for all incoming clients?
[19:36] sustrik yes
[19:37] Joes who will be doing queuing of the messages for temporary disconnected clients?
[19:37] sustrik the server
[19:37] Joes I see
[19:37] sustrik to be precise, the 0MQ socket in the server
[19:37] Joes hm
[19:38] sustrik all you need to do is set the ZMQ_IDENTITY socket option on the client
[19:38] sustrik that way the server socket would know that is should not deallocate the connection when it accidentally breaks
[19:38] sustrik the messages will be stored
[19:38] sustrik and once the same client re-connects they will be sent
[19:39] Joes that's the problem - I will have to write proxy between browser and 0MQ server
[19:39] mikko i think you are talking about separate issue
[19:39] mikko in this case:
[19:39] mikko chat server <-- 0mq --> broker <-- raw tcp --> client
[19:39] Joes yep
[19:40] Joes so, my question is - I'm adding extra logic on broker side to handle another wire protocol, message queuing (connection persistence logic like in 0MQ)
[19:40] Joes is it worth of the trouble?
[19:41] Joes broker can be "end point"
[19:41] Joes not scalable solution, though
[19:41] mikko the only thing you have described about the application this far that it's a chat like irc
[19:41] Joes yep, pretty much
[19:42] mikko is this from scratch or do you have something running already?
[19:42] Joes scratch
[19:42] Joes prototyping
[19:43] Joes well, in terms of possible load - ~12k users average online
[19:43] Joes each making ~1 message in 5 seconds (worst case)
[19:43] Joes ajax with long polling does not suit very well
[19:44] mikko why do you need to queue the messages if the client drops?
[19:44] Joes so, trying to find other possibilities apart of writing custom protocol on top of raw sockets
[19:44] Joes to keep state of the room and its recent history
[19:45] Joes it is doable by resending last N items on reconnect
[19:45] Joes which solves the problem, but.. if 0MQ provides it for free..
[19:49] mikko let me think for a sec
[19:49] mikko the biggest problem is the last part to the browser
[19:49] Joes yep
[19:49] Joes I want to get rid of it by having simple, yet functional, 0mq client
[19:50] Joes let's assume there's no WebSocket wrapping for now.
[19:50] Joes plain tcp sockets
[19:50] mikko you are not in control of the client side i assume?
[19:51] mikko as in installing 0mq on client :)
[19:51] Joes nope.
[19:51] Joes plain HTML+JS
[19:51] Joes and, maybe, some Flash.
[19:51] Joes (I already checked almost all language integrations - all of them are dependent on the native C library)
[19:54] mikko yes, i think you would need to implement some logic in your broker
[19:55] Joes in terms of code, for 1 incoming browser it will create 0mq socket or it will handle all incoming browser connections through 1 0mq socket?
[19:55] Joes what is better approach?
[19:56] Joes multiple sockets - free message queuing
[19:56] Joes one socket - easier overall architecture (and speed?), but manual queuing on the broker
[19:57] Joes plus, custom wire protocol from broker to browser
[19:57] Joes etc.
[19:57] mikko one socket also requires custom filtering i guess
[19:57] mikko you got messages coming in 1 socket destined to ~12k clients
[19:57] Joes well, I can read address from the message
[19:58] mikko so you need to write them to correct handles on other side
[19:58] Joes and then pick proper browser socket and send it there
[19:58] Joes yep
[19:58] Joes it = message
[19:59] mikko one socket might be easier to handle for events as well
[19:59] mikko otherwise you got a pollset of 12k items
[19:59] mikko that you need to maintain
[19:59] Joes yeah
[19:59] Joes bad idea.
[19:59] Joes ok, another idea.
[19:59] Joes I write pure JS implementation of the wire protocol
[19:59] Joes it's pretty simple
[19:59] Joes and broker will be dump WebSocket negotiator and TCP proxy
[20:00] Joes (forwards tcp connection to the real 0MQ server port)
[20:00] Joes I assume it might work.
[20:00] Joes each message sent will have address as a first chunk
[20:00] Joes so chat server knows who sent it
[20:00] Joes and, maybe, where
[20:01] Joes dump = dumb
[20:01] mikko i think i wrote websocket daemon ages ago using libev
[20:01] mikko it wasn't complicated
[20:03] Joes hm, someone already did it for Ruby
[20:04] Joes https://github.com/andrewvc/dripdrop
[20:04] Joes custom wire format from client
[20:04] andrewvc hey, that's me
[20:04] Joes browser that is
[20:04] Joes :-)
[20:04] Joes wonderful
[20:04] Joes there was pretty long discussion
[20:05] andrewvc oh?
[20:05] Joes but, to sum it up - any plans for pure JS 0mq client instead of custom protocol?
[20:05] Joes (or message format)
[20:05] Joes so, you can get rid of Ruby server and just relay messages to 0mq
[20:05] Joes straight
[20:05] andrewvc oh
[20:05] andrewvc well, there's a node JS project
[20:06] andrewvc and I was going to implement zmq_devices as part of ffi-rzmq as well
[20:06] andrewvc or do those options not cover it?
[20:06] Joes node JS is using native C library
[20:06] Joes I want something like DripDrop
[20:06] andrewvc oh
[20:06] andrewvc interesting
[20:06] Joes browser <-> 0mq
[20:06] Joes and simple TCP proxy
[20:06] Joes in between
[20:06] andrewvc well, dripdrop doesn't really require zeromq
[20:06] Joes which knows how to handle WebSocket handshake
[20:06] andrewvc it works well with websockets
[20:06] andrewvc and http
[20:06] andrewvc all evented
[20:07] andrewvc I'm a bit confused as to the actual question still I guess
[20:07] mikko andrewvc: but he wants to run this in browser
[20:07] mikko andrewvc: hence the pure js implementation i guess
[20:07] Joes yes
[20:07] andrewvc ohhh, well who would the browser talk to. Itself?
[20:07] Joes 0mq server, preferably.
[20:08] Joes here's another example: online real-time game.
[20:08] andrewvc oh, well, as far as I know you can't open a raw TCP conn from any browser still
[20:08] andrewvc so you're using websockets in the yes?
[20:08] Joes socketjs also works
[20:09] Joes (embeds small Flash object to proxy sockets)
[20:09] Joes gives plain sockets in JS
[20:09] andrewvc yeah, so, that's an interesting idea
[20:09] Joes I mean - sure, I can create custom protocol
[20:10] Joes write custom server software
[20:10] andrewvc what kinds of implementation were you guys thinking?
[20:10] Joes which will parse messages, etc
[20:10] Joes but I dont see a point
[20:10] andrewvc what would the api look like, in JS?
[20:11] Joes so, DripDrop already doing it - checked JS files, it is encapsulating messages into DD.Message
[20:11] andrewvc I think that's probably a good place to start, if you can throw up a gist for a simple case
[20:11] andrewvc yep
[20:11] andrewvc the JS prolly needs a little love at this point
[20:12] Joes so, imagine that there's no need to have custom wire protocol (json serialized) and JS would just talk to 0mq server using 0mq wire protocol
[20:12] Joes that's the simpliest case
[20:13] andrewvc hmmm, the 0mq wire protocol doesn't handle that much really
[20:13] andrewvc I mean, you could express multipart stuff, like socket identities for stuff like xreq/xrep
[20:13] Joes browser won't be server either
[20:13] andrewvc but I'm not sure how useful that is
[20:13] Joes sub or pub, mostly
[20:13] Joes maybe req
[20:14] andrewvc I'm off to grab some lunch, but it's an interesting idea
[20:14] andrewvc I'd love to see a gist of a proposed API
[20:14] Joes actually, from wire perspective it can be req
[20:14] Joes always
[20:14] Joes em, no.
[20:14] Joes I need to get updates from server.
[20:14] Joes anyway, I'll think about it.
[20:14] andrewvc yeah, I'd like to hear more
[20:15] Joes it won't be full-scale 0mq implementation anyway
[20:15] andrewvc If you could ping me at @andrewvc on twitter when you have something I'd be interested
[20:15] andrewvc anyway, off to lunch
[20:15] Joes k
[20:15] Joes bon appetit
[20:17] Joes Disconnected question - how do you handle security? If you don't want particular client to be connected, what you can do?
[20:17] Joes so, I don't want malicious client to DOS my message queue
[20:17] Joes can I forcibly disconnect him?
[20:17] Joes (in my application)
[20:20] mikko hmm
[20:20] mikko was searching for 0mq on twitter and most of the results are people using "0mq" instead of "omg"
[20:20] Joes :-)
[20:24] Joes mikko: any hints on how to forcibly drop connections on server side?
[20:25] Joes or 0mq not really suited for client-facing functionality?
[20:26] mikko there is really no way to forcibly drop a connection
[20:27] Joes ouch.
[20:27] mikko apart from closing the socket
[20:27] Joes on client side, I assume
[20:27] Joes ?
[20:27] mikko on the server-side but that closes all connections
[20:27] Joes oops.
[20:28] Joes means, 0mq not really suitable for client-facing stuff, as clients might misbehave
[20:28] Joes plus, if there's some sort of authentication
[20:28] Joes it's easier to disconnect client
[20:29] Joes than to ignore all his sent messages
[20:43] sustrik what's the difference? a malevolent client can reconnect immediately and sent another malformed message
[20:43] sustrik the only consequence is that the server would use much more CPU to handle the reconnecting instead of simply dropping the messages
[20:47] Joes yep
[20:47] Joes but
[20:47] Joes let's assume that it is pub/sub
[20:47] Joes with, let's say 10 subscribers
[20:48] Joes so, bad clients sends 1 message
[20:48] Joes and 10 messages total are sent to subscribers
[20:49] Joes effectively, network topology creates nice DOS environment
[20:49] Joes bad client sends*
[20:49] Joes I mean - there's no built-in security, so there's no way to filter these messages before they get into "internal" network
[20:50] Joes where workers sit
[20:56] Joes however, I think broker which will relay messages to internal network will work
[20:56] Joes and he'll do the authentication stuff too
[21:13] mikko Joes_: it sounds like a sensible approach

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy