IRC Log

Thursday March 10, 2011

[Time] NameMessage
[04:04] qw The 2nd video on http://www.zeromq.org/intro:read-the-manual has left channel audio problem
[04:05] Steve-o yup
[04:05] Steve-o could do with a filtering
[04:08] qw the top video was excellent
[04:10] Steve-o prod pieter when he's online later
[05:58] notostraca agh... I am trying to get lua to work with 0MQ, to no avail.
[05:59] notostraca I am on Windows 7 64-bit
[05:59] notostraca and I am not quite sure how to install the compiled libzmq dlls so that luarocks and/or cmake can find them
[06:00] notostraca Has anyone gotten lua to work with 0MQ on windows?
[07:26] datuanmac hello
[07:26] rem7 I was going over the guide, and on the topic of multithreading it says that you should create a context and pass that to the threads... but down in chapter 3 some of the examples are creating a context in their own threads... whats up with that?
[07:29] guido_g here the threads stand for processes
[07:29] datuanmac can you send to me link chapter 3 to refer ?
[07:30] guido_g rem7: the rule for real applications is: one context per process
[07:31] rem7 so if I'm not going to use inproc in the same application, its safe to create a context in every thread...?
[07:31] guido_g rem7: the rule for real applications is: one context per process
[07:34] rem7 ... confused... define process?
[07:34] guido_g not thread, a process
[07:35] uhu process == task
[07:36] guido_g http://en.wikipedia.org/wiki/Process_(computing) <- learn the basics
[07:38] rem7 re-wording from the first paragraph then... one context per application instance.
[07:39] uhu hi didn't know : a thread is part of a process/task
[07:39] guido_g omg
[07:41] rem7 lol
[07:42] datuan Are you integrate with system ?
[07:50] uhu system() or system
[08:01] datuan system
[08:49] larhat Hi, folks! Is it possible to implement zeromq as native library in some language and not as language bindings? i.e. implement zero-mq protocol on top of wrapped sockets. I thinks, it's possible, but i don't see any attempts to do it. am i missing smth?
[08:50] guido_g zeromq is mostly the high-speed implementation of the existing library
[08:51] guido_g the network-framing is described somewhere on zeromq.org
[08:52] guido_g i doubt that new implementations will comme close to the performance of this implementation in the foreseeable future
[08:55] larhat so, there isn't native library because it's not needed?
[08:56] guido_g native to what?
[08:56] larhat "native", yes. native to some language (ocaml, python, etc. )
[08:56] guido_g ah, no wouldn't make sense
[08:57] guido_g consider how slow this would be
[08:57] guido_g and how much trouble to maintain
[08:58] Guthur larhat: atm imo, zeromq is quite a good to produce a binding for
[08:59] Guthur quite a/quite
[08:59] guido_g thanks to the simple c api
[08:59] Guthur indeed
[08:59] guido_g i'm even tempted to use the c api in c++, much easier
[09:00] Guthur which means there would very little to be gained from implementing in another language
[09:06] Guthur At the weekend I had a strange notion of implementing a zeromq system in C++
[09:06] Guthur I soon realised live is too short for such pain
[09:06] Guthur life*
[09:14] uhu anyone here, who has exp. with smartsockets (talarian/Tibco)
[09:15] Steve-o I've heard of them
[09:15] uhu need pro/cons for discussion next week
[09:15] Steve-o aren't smartsockets eol?
[09:15] Steve-o smartpgm eol'ed a year or two ago
[09:16] uhu talarian invented long times ago, later tibco bought talarian
[09:16] Steve-o I'm a former TIBCO guy :D
[09:17] uhu u hv worked with them ?
[09:18] Steve-o I was the maintenance dev for TIB2MQ/TIB Adapter for MQ Series and Rendezvous 5
[09:19] uhu i see
[09:19] Steve-o nobody in Finance used smartsockets though, so I never touched it
[09:20] uhu are smartsockets not the base of the TIB ESB
[09:20] Steve-o shouldn't your comparison be with EMS?
[09:20] uhu just working for a big german logistics ...
[09:21] uhu now the plan a redesign of the whole stuff
[09:21] uhu the -> they
[09:21] guido_g if you need ømq know-how, i'm available :)
[09:21] Steve-o tibco ems is tibco's jaunt to TCP comms away from multicast - http://www.tibco.com/products/soa/messaging/enterprise-mesage-service/default.jsp
[09:22] Steve-o it came before the talarian purchase I think
[09:23] uhu our management prefer the oracle esb stuff
[09:23] uhu but i think, they don't know what they talking about
[09:23] Guthur does 0MQ run on any mainframe hardware/OS
[09:23] Steve-o s390
[09:24] Steve-o and vms
[09:24] guido_g uhu: you said it already "management"
[09:24] uhu is there a port on HP Nonstop ???
[09:24] Steve-o openvms
[09:25] Steve-o the dev is from HP, on the ML
[09:25] Steve-o http://zeromqonopenvms.blogspot.com/
[09:26] Guthur guido_g: do you maintain one of the language bindings?
[09:27] guido_g nope
[09:27] guido_g why?
[09:27] Guthur looking for advice on some course of action for EINTR on things like poll and recv
[09:28] Guthur I'm wondering whether clrzmq2 should try to deal with it, or leave it for the user
[09:28] guido_g uhhh... there was something going on for the python bindings
[09:28] Guthur oh ok, I'mm have a wee look at that
[09:28] Guthur I'mm/I'll
[09:29] guido_g http://lists.zeromq.org/pipermail/zeromq-dev/2010-September/005822.html
[09:29] uhu sorry, i know ... the stoneage is over ... but our company is still using this machines
[09:30] guido_g hey, the ise in ny is still using vms too i think :)
[09:32] Steve-o well at least they have a tcp/ip stack
[09:32] pieterh hi guys
[09:32] Guthur guido_g: I think that is fixed now though, because I am getting -1 errno=EINTR
[09:33] guido_g Guthur: the discussion was lengthy, i don't remember it
[09:33] guido_g pieterh: howdy
[09:33] Guthur I'm in two minds what to do now
[09:33] guido_g Guthur: but i think in certain cases the binding should handle eintr
[09:34] pieterh hi guido_g
[09:35] Guthur guido_g: I was considering trying to handle it in the binding
[09:35] Guthur would make life simpler for the user
[09:36] guido_g Guthur: i can have a look at the python binding, might be that there is some help
[09:36] Guthur guido_g: cool
[09:40] guido_g so far no eintr found...
[09:43] Guthur from the way I am understanding the code, it would not matter to pyzmq
[09:44] Guthur it just returns a list of r/w able sockets
[09:44] guido_g no, pyzmq.core will deliver upwards
[09:44] guido_g *deliver it
[09:45] Guthur ok
[09:46] guido_g Guthur: you're on the low-level bindings atm?
[09:50] Guthur i'm maintainer for clrzmq2, and I'm hoping to improve the handling of this issue there
[09:50] Guthur It just binds the low level C lib
[09:50] Guthur was that your question?
[09:50] Guthur or do you mean the pyzmq
[09:51] Guthur I was looking at https://github.com/ellisonbg/pyzmq
[09:51] guido_g i meant if your're currently working on the lower levels of the binding or some language dependant abstractions on top
[09:52] guido_g nahh... the official one is https://github.com/zeromq/pyzmq
[09:52] Guthur oh, ok
[09:53] guido_g Guthur: if you provide the low-level api in clrzmq2, it should simply signal that an EINTR happend, same as the C version
[09:53] Guthur yeah that's what happens at the moment
[09:53] Guthur it throws an exception
[09:53] Guthur so the user can catch it if they wish
[09:54] guido_g if you have some more .net like abstractions on top of that, it's a matter of the overall architecture of the environment, i'd say
[09:54] guido_g should be fine
[10:23] Guthur this experience with EINTR has made me think I don't like linux's signalling mechanism much
[10:24] Guthur it can be a bit of PITA
[10:24] guido_g it's not only linux
[10:27] Guthur implementing the later guide examples in C# has definitely proved more challenging than it should have been.
[10:28] Guthur no IPC on windows, and signalling headaches on POSIX OSs
[10:40] Guthur ...really need to get working on that IPC for windows stuff
[10:44] Guthur is there 2.2 roadmap?
[10:44] Guthur pieterh: you mentioned it on the mailing list, but I don't see it
[10:44] Guthur oh nvm
[10:44] Guthur found it
[10:45] pieterh Guthur: sorry about that
[10:45] pieterh we need to get IPC working better IMO
[10:46] pieterh even the Linux / POSIX implementation is sucky
[10:46] pieterh I'd like to make a portable IPC based on shared memory
[10:47] pieterh Guthur: did you get any further with those signaling issues?
[10:47] Guthur pieterh_: yeah, it's just something that will happen on POSIX
[10:47] pieterh ... I'm using Linux all the time, have never hit it
[10:48] pieterh or am just somehow ignoring it?
[10:48] Guthur it's more likely to happen on Mono due to the GC using signals
[10:48] pieterh more likely vs. never happens, I'd say it's a Mono issue
[10:49] pieterh thus perhaps solvable
[10:49] Guthur yeah, it's a catchable exception, I was in two minds whether to try and solve it in the clrzmq binding or leave it for the user to deal with
[10:50] pieterh would you summarize what you know about it, send that to the list?
[10:50] Guthur sure, I'll do that when I get home from work
[10:50] pieterh IMO it's not something the user should see, it's has no sense within 0MQ
[10:51] pieterh making the examples is in any case like doing 0MQ University :-)
[10:51] Guthur pieterh_: it actually appears that it never used to be in 0MQ
[10:51] Guthur it was added
[10:51] Guthur http://lists.zeromq.org/pipermail/zeromq-dev/2010-September/005822.html
[10:52] pieterh ah, this is the famous Ctrl-C handling
[10:52] pieterh yes, signals should be passed up to the binding, however...
[10:53] pieterh it seems abnormal that you are getting EINTR *all the time*
[10:54] pieterh brb, need to switch message apps, this one is terrible...
[10:55] guido_g hehe
[10:57] pieterh_1 re
[10:59] guido_g wb
[10:59] guido_g w/ different color, how nice :)
[11:02] pieterh_1 I think the colors are entirely in the imagination of your IRC client :-)
[11:03] guido_g hmmm
[11:06] Guthur pieterh_1: it seems to be at startup that the Mono runtime is generating some signals that were bubbling up when I was trying to Poll or Recv
[11:06] Guthur after a talk with a Mono Dev I am of the understanding that it may be related to the GC
[11:07] pieterh_1 Guthur: that sounds plausible
[11:09] pieterh_1 those signals shouldn't appear to 0MQ users, obvously
[11:11] Guthur pieterh_1: would you be of the opinion that I should try to deal with EINTR in the clrzmq binding
[11:11] sejo can you send a smq message without using smq? is there a way to recontruct a message?
[11:12] pieterh_1 Guthur: if it's Mono generating random EINTRs, I'd probably take a large stick to that developer until he fixes it
[11:12] pieterh_1 if it's other signals you can filter them
[11:13] pieterh_1 sejo: what exactly do you mean by "smq message"?
[11:13] sejo zmq sorry
[11:13] pieterh_1 how would you send a zmq message without zmq?
[11:14] pieterh_1 and why?
[11:14] sejo well i'm still stuck with the twisted loop and trying to figure out how to fix it
[11:14] Guthur pieterh_1: i think it's due to 0MQ being in a blocking syscall, and so the signal will cause that to return with EINTR
[11:14] Guthur my understanding may be flawed though
[11:15] pieterh_1 Guthur: yeah... but if you know what signals the GC is generating you can set up your own handler for them, so 0MQ never sees them
[11:15] Guthur ah ok
[11:15] pieterh_1 bleh, I'm confusing two things
[11:16] pieterh_1 EINTR is the errno after an interrupted system call, it's not the signal
[11:16] pieterh_1 Ctrl-C would be SIGINT for example
[11:16] pieterh_1 so IMO you want to discover what actual signals the Mono GC is generating, and then set up handlers to ignore them
[11:17] pieterh_1 then 0MQ's system calls won't be interrupted by *those* signals
[11:20] sejo pieterh: so I'm trying to get messages into a queue. but when pushing from twisted it slows down my reactor
[11:22] pieterh_1 sejo: is there anyone else using twisted with 0MQ?
[11:23] sejo no, there was a poc on github, but not working
[11:23] pieterh_1 ok, so I know nothing at all about twisted, however...
[11:24] guido_g sejo: where does the slowdonw happen? care to show the code?
[11:24] pieterh_1 this is Python, there is a large community of pyzmq users
[11:24] pieterh_1 make a minimal test case, publish that, discuss on zeromq-dev and here
[11:24] sejo guido_g: each time you do a blocking io, the twisted main_reactor waits on it
[11:25] pieterh_1 you do need to come up with code if you want help on specific problems
[11:25] guido_g this is why it's call blocking
[11:25] guido_g *called
[11:25] sejo that's why twisted is used to circumvent the blocking io as much as possible
[11:25] guido_g simply don't use blocking call then...
[11:25] sejo yeah but there is no way to do a nen-blocking connection to zmq, or I should use a basic socket protocol that already exists
[11:26] guido_g w/o code, it's all guessing
[11:26] sejo guido if I use the smq.send_json it blocks the reactor in such way i only can send one a sec (I crawl now 5-10 urls/s)
[11:26] sejo if I add the seromq.send_json it goes to 1/s
[11:26] sejo zmq
[11:27] pieterh_1 sejo: it's not 'slow'
[11:27] sejo pieterh_1: i know zmq isn't slow
[11:27] sejo and the twisted neither, it's combining them that creates 2 diffrent eventloops and they don't work together imho
[11:28] sejo which is logic
[11:28] pieterh_1 no, I mean it's not 'slowing down', you're using it wrong
[11:28] pieterh_1 and without seeing an example of your code, this discussion is fruitless
[11:28] sejo let me paste an example
[11:28] pieterh_1 sejo: yes, please, a working minimal example
[11:29] guido_g sejo: use non-blocking calls then
[11:46] sejo http://dpaste.com/485798/
[11:46] sejo sending
[11:46] sejo http://dpaste.com/485800/
[11:46] sejo receiver and sender to the workers
[11:47] sejo brb food
[11:48] guido_g sejo: read the guid and ahve a look at the pyzmq examples
[11:48] guido_g sejo: this all weird
[12:01] sejo I did I used the taskvent/tasksink/taskworker examples from the python guide
[12:03] guido_g sejo: your code is broken, please try to get the ømq basics first
[12:15] sejo meh
[12:18] sejo not wanting to be rude but i don't see what I'm doing wrong if I look at these examples: https://github.com/imatix/zguide/blob/master/examples/Python/taskvent.py https://github.com/imatix/zguide/blob/master/examples/Python/taskwork.py
[12:18] pieterh_1 sejo: in sender, what's the lifespan of the context object?
[12:18] sejo pieterh_1: the time of a full crawl +- 5000 messages
[12:19] pieterh_1 where is recv_json defined? is that a standard pyzmq method?
[12:19] pieterh_1 hmm, presumably so...
[12:19] sejo yes
[12:19] sejo the couchdbkit people showed me that :p
[12:20] pieterh_1 where is the reactor in all of this? and twisted?
[12:20] pieterh_1 this is not complete code, there's a lot missing
[12:20] sejo pietthe reactor is basicly hidder from my spider
[12:20] sejo so that's in the back
[12:21] sejo the process_response is a callback
[12:21] pieterh_1 look, could you please make a *working minimal example*
[12:21] pieterh_1 do you understand what I mean when I say that?
[12:21] pieterh_1 fragments of code are not useful
[12:22] sejo actually then I need to provide the full scrapy code... nvm
[12:22] pieterh_1 you are not understanding IMO
[12:22] pieterh_1 this is not us being difficult, it is an essential step in your own understanding of the problem
[12:22] pieterh_1 look...
[12:22] sejo i understand also
[12:23] pieterh_1 you use 0MQ + twisted + whatever within a complex environment
[12:23] pieterh_1 things do not work
[12:23] pieterh_1 so far, all normal
[12:23] pieterh_1 your first step is to isolate your 0MQ code from the rest
[12:23] pieterh_1 minimal working test case
[12:23] pieterh_1 if the problem goes away, you know immediately that it was in whatever you removde
[12:23] pieterh_1 *removed
[12:23] pieterh_1 if the problem is still there, you can ask for help
[12:24] pieterh_1 no-one here is going to help you if you don't take this first step
[12:24] sejo pieterh: i understand
[12:40] Guthur pieterh_1: revisiting the EINTR issue, I think the blocking syscalls will always return EINTR if there is a signal, no matter if a handler is set or not
[12:40] pieterh_1 have you tried it, or are you speculating?
[12:41] Guthur pieterh_1: going by how the spec reads
[12:41] pieterh_1 what spec?
[12:41] Guthur man page for poll
[12:41] pieterh_1 zmq_poll?
[12:41] Guthur no linux poll
[12:41] Guthur the syscall
[12:41] pieterh_1 ok
[12:41] Guthur the GC does use two signals
[12:41] Guthur but it should catch them itself
[12:42] pieterh_1 should, doesn't, I assume
[12:42] pieterh_1 what signals are they?
[12:42] pieterh_1 all signals can be trapped except SIGKILL and SIGSTOP
[12:42] Guthur it does, but if my understanding is correct during some blocking syscalls such as Poll and Recv it can't catch them
[12:43] pieterh_1 sigh
[12:43] pieterh_1 system calls are not special magic
[12:44] pieterh_1 thus the Mono GC is broken and not properly handling those signals
[12:44] Guthur Poll: A signal occurred before any requested event. Recv: The receive was interrupted by delivery of a signal before any data were available.
[12:44] Guthur from the man page ^
[12:44] Guthur that's for EINTR
[12:44] Guthur Poll is the most telling
[12:44] pieterh_1 Guthur: I can read man pages too, these sentences do not say "you cannot catch signals during a poll"
[12:45] Guthur Poll says if it occurs
[12:45] pieterh_1 they say, "a signal (THAT YOU DID NOT HANDLE) arrived"
[12:45] pieterh_1 "delivery of a signal"
[12:47] Guthur I can run the code later with the GC turned off
[12:47] pieterh_1 try "signal (whatever-signal-mono-uses, SIG_IGN)" if you can
[12:48] pieterh_1 file a bug report with the Mono people, this is not a 0MQ issue afaics...
[12:48] pieterh_1 perhaps note in the C# binding page, "Mono is broken, see issue #xyz here"
[12:49] Guthur Certainly not a 0MQ issue
[12:50] Guthur I'd just like to deal with it nicely in the clrzmq binding
[12:50] Guthur if possible
[12:50] pieterh_1 if (running on Mono)
[12:50] pieterh_1 exit "friends don't let friends use this rubbish"
[12:50] pieterh_1 endif
[12:51] pieterh_1 I'd seriously choose another language if you want portability
[12:51] Guthur http://www.ustash.com/mono/Embedding_Mono.html#Signal_handling
[12:51] Guthur that's the signals Mono
[12:51] Guthur uses
[12:52] Guthur pieterh_1: I'm the clrzmq2 maintainer, choosing another language is not really an option in that capacity
[12:52] pieterh_1 :-)
[12:52] pieterh_1 So harass the Mono developers until they fix this
[12:53] pieterh_1 You can't work around it because you have no way of knowing what signal actually interrupted poll
[12:54] pieterh_1 If you ignore EINTR systematically your applications won't exit on real signals like SIGINT
[13:30] Guthur pieterh_1: It would appear that Poll and Recv will return EINTR if there is a signal
[13:31] Guthur I was talking to someone on ##kernel
[13:31] Guthur and he mentioned that SA_RESTART can be used to restart the syscall
[13:32] pieterh_1 did you specifically ask whether this applied to handled signals or unhandled signals?
[13:32] Guthur or rather it wont return -1
[13:32] Guthur yeap
[13:32] Guthur oh wait, bugger
[13:32] Guthur I meant to
[13:32] Guthur two secs
[13:32] pieterh_1 we do know what the man page says
[13:33] pieterh_1 please ask this, specifically:
[13:33] pieterh_1 "if I raise a user-defined signal which I've set SIG_IGN for, will that interrupt poll?"
[13:34] Guthur it should be able to mask them out if that is what you mean by SIG_IGN
[13:35] Guthur When a system call is slow and a signal arrives while it was blocked, waiting for something, the call is aborted and returns -EINTR, so that the library function will return -1 and set errno to EINTR. Just before the system call returns, the user program's signal handler is called.
[13:35] pieterh_1 SIG_IGN has a specific meaning, read the signal() man page
[13:35] Guthur that is taken from here http://www.win.tue.nl/~aeb/linux/lk/lk-4.html
[13:37] pieterh_1 yes
[13:37] pieterh_1 ok, makes sense
[13:37] pieterh_1 the handling of the signal can be set to SA_RESTART
[13:37] pieterh_1 meaning the system call will continue
[13:38] Guthur yeah, I should check with the Mono Devs to see if they do that
[13:39] hoschi Hi, how can I do RPC with zeromq when I have different services (aka REP)? Should I use another socket combination, or must I write a address-based router like in the manual?
[13:40] Guthur hehe, I was never expecting to have dive into Syscalls and Signals when developing this binding
[13:46] pieterh_1 it's surprising... but we learn something new every day
[13:47] pieterh_1 hoschi: how much of the Guide have you read and digested?
[13:48] hoschi pieterh_1: I read chapter one and had a look at chapter 3
[13:48] hoschi I also read the introduction blog post from nicholas
[13:48] pieterh_1 so please read all four chapters, and take the time to understand what they say
[13:48] pieterh_1 thanks
[14:12] Guthur is sustrik still in the US
[14:27] pieterh_1 Guthur: afaik he's traveling back about now
[14:39] private_meta So he finished his San Francisco Meetup? :D
[14:40] Guthur yep, sounds like it was a great success
[14:41] Guthur I hope there will be similar setup on this side of 'pond' sometime
[14:41] pieterh_1 Next week, in London
[14:42] Guthur yeah, i kind of mean a more formal setup, a little like Martin mentioned
[14:42] Guthur personally I'm not overly interested in trip to the pub
[14:42] Guthur personal thing
[14:42] pieterh_1 You're in London?
[14:43] Guthur nope, but I am in the UK
[14:43] pieterh_1 well, if you can find an "more formal" place in London, for next Wednesday around 5-6pm, we can do that
[14:44] pieterh_1 it's not really about hope, more about making it happen... :-)
[14:44] Guthur very true
[14:44] pieterh_1 I'm there all day, happy to be anywhere in London
[14:44] Guthur next time maybe, it's a little short notice now for me
[14:45] pieterh_1 this is the 21st century...1
[14:45] pieterh_1 !
[14:45] Guthur tell that too my employer
[14:45] pieterh_1 "employer"...?
[14:45] pieterh_1 sorry to hear that
[14:45] pieterh_1 :-) just kidding
[14:45] Guthur I'm technically suppose to give about 2 weeks notice
[14:46] pieterh_1 to attend a technical seminar in the evening?
[14:46] pieterh_1 your employer should be paying your costs...
[14:46] Guthur ah yes, well when I say the UK, I mean Northern Ireland
[14:46] pieterh_1 sigh
[14:46] Guthur it does require a little more logistics, hehe
[14:47] Guthur I sometimes have business trips to London though
[14:47] Guthur our main regional office is there
[14:48] pieterh_1 fair enough
[14:48] pieterh_1 if and when you get the opportunity to find a formal location in London, we can do a meetup
[14:48] pieterh_1 I'll make sure I get there
[14:48] Guthur cool
[14:48] Guthur are you going next wednesday
[14:48] pieterh_1 for sure, it's in a pub!
[14:49] Guthur hehe
[14:49] guido_g *sigh*
[14:49] pieterh_1 personal thing... :-) anyhow, they don't actually have *beer* in the UK
[14:49] guido_g guinness would do :)
[14:50] pieterh_1 guido_g: I'm going to have to smuggle some real beer across from Brussels with me
[14:50] guido_g hehe
[14:50] Guthur what about coffee shops, I do like coffee
[14:51] pieterh_1 coffee shops are great, but not after 5pm or so
[14:51] pieterh_1 I suspect the timing of this meetup is "after work", easier for most people
[14:53] pieterh_1 Guthur: how about a real event, in Brussels, later in the spring?
[14:54] pieterh_1 a proper 0MQ seminar / workshop
[14:55] Guthur I'd be up for that
[14:55] Guthur I haven't been to brussels before
[14:56] Guthur Sounds a great idea actually, imo
[14:56] pieterh_1 I'm wondering whether there'd be demand for a proper seminar session
[14:56] pieterh_1 independently of a meetup later on
[14:56] pieterh_1 seminar = your company pays $$$ to send you
[14:57] guido_g shit
[14:57] guido_g i'm my company
[14:57] pieterh_1 guido_g: ah, but there is also an organizing team
[14:58] guido_g ok
[14:59] pieterh_1 eclipse, is that like an emacs clone but Java instead of Lisp?
[14:59] guido_g *NO*
[14:59] guido_g emacs works
[14:59] pieterh_1 lol
[14:59] guido_g my $HOME is where I lay my .emacs
[15:00] pieterh_1 vi is much better
[15:00] guido_g matter of taste
[15:00] guido_g i do use vi
[15:00] guido_g strangely, when i edit cofnig files etc. i use vi
[15:01] guido_g or when i'm root
[15:01] pieterh_1 yeah, anything system feels more natural using vi
[15:01] pieterh_1 it's supposed to look ugly and be nasty
[15:01] guido_g matter of age :)
[15:01] pieterh_1 OK, I'm designing a simple name service for 0MQ
[15:01] pieterh_1 going to be the last example in Ch4
[15:01] guido_g cool
[15:02] pieterh_1 the idea is client-service reliability with no broker
[15:02] guido_g i'm on writing a zmq reactor/ioloop in c++
[15:02] pieterh_1 clients connect to N services then do a kind of paranoid pirate on them
[15:02] pieterh_1 hmm, would you care to document that upfront in a language-neutral fashion?
[15:03] guido_g uh
[15:03] pieterh_1 I'd like to make a reactor as an example for later in the guide
[15:04] guido_g it's just a glorified zmq_poll
[15:04] pieterh_1 (a) take zmq_poll, (b) glorify it.
[15:04] pieterh_1 ok, that's clear
[15:04] guido_g *sigh*
[15:07] guido_g https://github.com/guidog/cpp/blob/master/zmqcpp/reactor.cpp <- example/test how it could work
[15:09] guido_g but i'm not sure if this is the way to go in/with c++
[15:09] pieterh_1 i assume you also want timer events
[15:10] guido_g again?
[15:12] Guthur emacs rocks
[15:13] guido_g yeah
[15:13] pieterh guido_g: yeah
[15:54] private_meta quitting a client because of a computer switch... preposterous
[16:08] guido_g pieterh: reactor/ioloop page https://github.com/guidog/cppzmq/wiki/ioloop
[16:11] pieterh guido_g: hmm, what I miss in these models is any kind of FSM
[16:11] pieterh i.e. "if current socket state is S and I get event E then execute steps F1, F2, F3"
[16:12] pieterh "however if state is S2 then execute F5"
[16:12] pieterh "and ignore event in any other state"
[16:12] pieterh etc.
[16:12] guido_g that would mean to track the sockets and their states
[16:12] pieterh sure
[16:12] guido_g because ømq doesn't give us the required information
[16:13] pieterh np, I'll make a FSM reactor pattern at some stage
[16:13] pieterh it's not hard
[16:13] pieterh just unusual
[16:13] guido_g the hard part is to track the socket states reliably
[16:13] pieterh you don't need to
[16:13] guido_g explain
[16:14] pieterh in the FSM definition, a given state/event transition always results in a specific next state
[16:14] guido_g the finite part
[16:14] pieterh socket "state" is not a 0MQ concept, it's an application concept
[16:14] guido_g aahhhh
[16:15] pieterh imagine implementing MDP like this
[16:15] pieterh sure
[16:15] guido_g it's more a general fsm thingie then
[16:15] pieterh yes
[16:15] pieterh but hooked into zmq sockets
[16:15] guido_g sorry, won't do that, not related to ømq :)
[16:16] pieterh but hooked into zmq sockets
[16:16] Guthur that's focus for you
[16:16] Guthur dogged even
[16:16] pieterh very dogged
[16:16] guido_g pieterh: i'd say layered on top of something that can deal w/ ømq socket
[16:16] guido_g +s
[16:17] pieterh weird, this is what I'm talking about
[16:17] pieterh it's like you read my mind...
[16:17] guido_g +sigh*
[16:17] pieterh oh, wait, I just said this, yeah :-)
[16:17] guido_g you said "hooked into", which imo is not the way it'll work
[16:18] guido_g the sockets are just another event source in this case
[16:18] pieterh if you *really* want to know...
[16:18] pieterh go find an old project called Libero
[16:18] pieterh there are more modern FSMs but I like the Libero FSM model, it's simple
[16:20] guido_g code generation...
[16:20] guido_g hmm hmm hmm
[16:20] pieterh that's how Libero does it, yes, but it's not mandatory
[16:21] guido_g allone the task of inventing a dsl for this daring
[16:21] guido_g *this is
[16:21] pieterh dsl?
[16:21] guido_g domain specific language
[16:21] pieterh ah, :-)
[16:22] Guthur crack out the Lisp
[16:22] guido_g or how to express the fsm in a programming lanugage
[16:22] Guthur only way to go for DSLs
[16:22] pieterh no, these days we just use XML or JSON or s-expressions or whatever
[16:22] pieterh trivial, and we have generic code generators for (e.g.) XML which are infinitely more powerful than Libero
[16:22] guido_g http://www.mcternan.me.uk/mscgen/ <- nice dsl :)
[16:22] pieterh if you like code generation, check out GSL4 on github
[16:22] guido_g no thanks
[16:23] pieterh the DSL is as such irrelevant as long as it has structure
[16:23] Guthur I'd rather tear my hair than use XML as a language
[16:23] guido_g i'm the only code generator i trust (sometimes)
[16:23] Guthur hair out*
[16:23] pieterh sure, shrug
[16:23] guido_g Guthur: almost done :)
[16:29] nadime Anyone know if it's an issue to set up pollitems in multiple threads? (but only run zmq_poll in a threadsafe manner)? getting weird behavior.
[16:29] guido_g we need to stop pieterh somehow, otherwise the ch4 of teh guide will become a tome in itself ,)
[16:30] nadime sorry let me be clearer: these pollitems in other threads share a socket
[16:30] pieterh ssh, don't tell him...
[16:30] pieterh nadime: sharing the same socket from multiple threads?
[16:30] guido_g nadime: sounds weird, "don't do this" i'd say
[16:31] nadime yeah, so I have 10 worker threads. 10 pollitems. they all share 1 subscribe socket. However,, i only run zmq_poll in a threadsafe manner (i.e. only 1 thread at a time polls, then receives anything that needs to be received, then releases its memory barrier).
[16:32] pieterh nadime: how much of the Guide have you read?
[16:32] nadime all of it
[16:32] nadime i have no choice to do what i'm doing unless i completely rearchitect using XREQ/REP.
[16:32] pieterh did you read the part where it says in big letters, "DON'T SHARE SOCKETS BETWEEN THREADS!"
[16:32] pieterh like several times
[16:32] nadime ok, but then you made it possible to do so in 2.1?
[16:32] guido_g omg
[16:32] nadime what did you expect, people not to use functionality that's there?
[16:33] pieterh yeah, which I regret we even documented
[16:33] guido_g nadime: you've been warned, so...
[16:33] pieterh because people systematically do what you do, think "Hey, this is cool" and then...
[16:33] pieterh weird stuff happens
[16:33] pieterh we get this all the time, literally
[16:33] pieterh "hey dudes, weird stuff happens!?"
[16:33] pieterh "stop sharing sockets between threads"
[16:33] pieterh "hey, how did you know...?"
[16:34] pieterh the functionality was added for low-level work such as in language bindings
[16:34] pieterh I'm really sorry, but sockets are not thread safe
[16:34] nadime I understand, and perhaps I shouldn't bother using zeromq, but you do get that it's a big pain in the ass not to be able to share sockets between threads, right?
[16:34] pieterh I get it
[16:34] nadime I can give you countless examples, and you can certainly rearchitect them to not share sockets
[16:35] nadime but it requires annoying rearchitecture
[16:35] pieterh but if you make sockets threadsafe (inside 0MQ)
[16:35] pieterh you basically make a different product
[16:35] pieterh which is slow, locky, but easier to use
[16:35] nadime for example, designing a generic messaging library for other people to use without sharing sockets between threads (i.e. similar to your language binding example)
[16:35] guido_g nadime: why did you architect it that way, then?
[16:36] cremes nadime: no need to redesign
[16:36] cremes put a FORWARDER device in the thread handling the polling
[16:36] pieterh indeed
[16:37] cremes have each worker thread subscribe to the forwarder
[16:37] pieterh use 0MQ, don't fight it
[16:37] cremes piece of cake; this is a great pattern
[16:37] nadime yes, i can do that, and it will likely be what i end up doing, but the point is that it's (a) extra latency, (b) if you say you now can migrate sockets, then it should be ok!
[16:38] pieterh (a) it's not, it's inproc and near zero latency
[16:38] pieterh (b) 'migrate sockets using full memory barriers' is not the same as 'share from 10 threads at once'
[16:38] pieterh sorry, please do read the man page carefully
[16:38] cremes right, inproc is a few microsecons
[16:38] pieterh migration means you move the socket from thread A to thread B, where you then use it
[16:39] pieterh what you are doing is rather different afaics
[16:40] pieterh (c) if you care about latency, you don't use shared state concurrency, period
[16:41] pieterh (d) this is all covered in the Guide, isn't it?
[16:42] cremes pieterh: i don't know if telling people to read the guide *again* is a good answer
[16:42] cremes i think nadime is doing the right thing here
[16:42] nadime (a) isn't inproc implemented with tcp on windows?
[16:42] cremes he's brining up a challenge and asking "why doesn't this work?"
[16:42] pieterh cremes: he's not explaining his use case
[16:42] nadime (b) i will look at the documentation again, but i believe 2.1+ claims to be threadsafe as long as you are (which is another story)
[16:42] cremes nadime: no, ipc uses tcp, inproc is flipping pointers
[16:42] pieterh nadime: next time please explain what you want to do, ask how to do it, if you don't see the obvious answer
[16:42] nadime (c) that's not totally true, but a longer argument
[16:42] cremes pieterh: that's a common problem :)
[16:43] nadime (d) see (b)
[16:43] pieterh complaining about things not working as you expect is pointless unless you are quite expert in 0MQ and your expectations already work the right way
[16:43] nadime let me give you an example that's annoying and you can tell me if i'm missing something stupid
[16:43] pieterh go for it
[16:43] cremes the docs should probably be changed to say that sockets have enough thread safety to allow
[16:43] pieterh cremes: sigh
[16:43] cremes language bindings to safely handle them from gc threads, etc
[16:44] pieterh nope, the docs should say, "if you share sockets you deserve the worst you get"
[16:44] cremes but they should *not* be used from multiple threads as a design pattern
[16:44] cremes pieterh: sure, that works too
[16:44] nadime i create a log server which receives a bunch of messages from anyone who cares to tell it anything.
[16:44] pieterh with a footnote saying, "if you're a language binding author, contact us offlist"
[16:44] nadime to go along witht his log server, i make a "logger" that instantiates a class capable of logging to this server.
[16:44] pieterh nadime: so far so good
[16:44] nadime all you do is run logger.log(<msg>) and it logs it for youl.
[16:44] cremes nadime: go on... i wrote this exact thing recently...
[16:45] nadime now let's say my friend bob uses my logger library. he basically has to create one logger per thread
[16:45] pieterh yes
[16:45] cremes yep
[16:45] pieterh that costs... nothing... so great
[16:45] nadime of course it costs something
[16:45] pieterh not as much as this conversation
[16:45] pieterh I'm serious
[16:46] cremes pieterh: come on
[16:46] cremes nadime: anything more?
[16:46] pieterh sigh, provide me with a measurable significant cost, not "non zero"
[16:46] pieterh nadime: apologies
[16:46] nadime i don't understand, i'm not arguing your software is useless, obviously i think it's great
[16:47] pieterh you have a design that works, ok?
[16:47] nadime i'm pointing out an issue, if the gain is greater than the loss, great
[16:47] pieterh each thread creates a logger, has its own sockets, etc.
[16:47] pieterh that is the brute force design, obvious, simple, scalable
[16:47] cremes nadime: here is how i solved this *exact* case
[16:47] pieterh you can break the threads into their own processes, it still works
[16:47] pieterh right?
[16:47] nadime sure, but let's say you use a threadpool
[16:47] pieterh now you believe you have a cost issue
[16:47] nadime now the function you call within the pool has to create a new logger
[16:47] pieterh and you try to optimize that away
[16:47] cremes each thread has a pub socket
[16:47] pieterh and you instead create a much larger design and complexity cost
[16:48] cremes the log server, which handles writing the messages to persistent storage, has a sub socket
[16:48] cremes i make sure to use a single context so that i can communicate via inproc transport (no tcp or ipc overhead)
[16:48] nadime right, and that's easy using a factory
[16:48] cremes the log server binds to the well-known port first (important for inproc)
[16:49] cremes each thread instantiates a logger object (a few hundred bytes) and connects to the well-known port
[16:49] cremes done
[16:49] cremes Guthur: it's happening async! :)
[16:49] Guthur ZeroMQ is the answer then
[16:50] nadime no question, i'm not saying it's that hard. i'm just saying it's a really, really different paradigm to say that if you make a library that uses zeromq it's a once per thread library
[16:50] pieterh ah, paradigms... why didn't you say so?
[16:50] pieterh you started with "I'd need to rearchitect using XREP sockets"...
[16:50] nadime well no, i have two possible solutions to be fair
[16:50] nadime use a forwarder as cremes suggested
[16:51] nadime or use XREP to evenly distribute messages
[16:51] pieterh hang on, nadime, please explain something missing here
[16:51] nadime sure?
[16:51] pieterh you have N clients sending to 1 logger, right?
[16:51] pieterh that is one half of your problem
[16:51] pieterh you then have 1 logger talking to N workers, is that so?
[16:51] pieterh could you please explain that, I missed it somehow
[16:52] nadime so the logger actually doesn't migrate sockets, i just gave you that as an example. my more complicated multithreaded app is a market data distributor
[16:52] nadime let me explain it in more detail
[16:52] pieterh please do explain the use case, otherwise it's like asking for a gallon of gas but driving up an hour later
[16:53] nadime i have a server which receives multiple market data feeds and distributes them via pub to clients which use a single sub socket.
[16:53] pieterh sure
[16:53] pieterh that's it?
[16:53] nadime my data server will handle historical data as well as realtime, and for that i want to pump it up to as fast as possible
[16:53] nadime no, hold on
[16:53] nadime so i'm operating right now at something like 100x realtime on level 2 data
[16:53] nadime which means a lot of messages
[16:53] pieterh sure
[16:54] nadime it's soemthing like 4 million/sec
[16:54] nadime zeromq is fantastic. it's working very well.
[16:54] pieterh continue...
[16:54] nadime the problem i have is that on my client, i.e. the data receiver, i offer a callback functionality
[16:54] nadime i.e. when the client gets message-type X, it runs a callback inline
[16:54] nadime if the callback isn't lightning fast, that can slow down message receipt
[16:55] pieterh sure
[16:55] nadime which at 4 million/sec blows things up
[16:55] pieterh sure
[16:55] nadime so
[16:55] nadime i wanted to create multiple threads so that another thread can be receiving a message while the other threads are running the callback
[16:55] nadime now let me point out
[16:55] nadime this is a case where i can easily change this to support not migrating the sockets
[16:56] pieterh hang on, nadime
[16:56] pieterh publisher and subscribers are all on different boxes?
[16:56] nadime yes
[16:56] pieterh so one subscriber has a large multicore box to play with
[16:56] nadime all of these boxes are pretty powerful, yep
[16:56] pieterh how many cores? just for info...
[16:57] nadime server has 12, clients have 6
[16:57] nadime that's physical
[16:57] cremes a forwarder device isn't even necessary in this case
[16:57] pieterh ok, great
[16:57] pieterh so imagine you have a blank sheet, and can design this from scratch
[16:58] nadime k
[16:58] pieterh you have one app thread subscribing and pulling down messages as fast as it can
[16:58] pieterh you may be able to filter, conflate, etc. but that's not important
[16:58] pieterh every now and then it decides to invoke a callback for a message
[16:59] pieterh the callbacks run as separate 0MQ threads, each waiting on a REP socket
[16:59] pieterh you can use any of a number of patterns here, including XREP routing to those callback threads
[16:59] pieterh you use inproc, of course
[16:59] nadime yes, and that's a great design, again i'm sort of not being clear here, and i apologize .. i know i can rearchitect this without that much trouble, it will probably take me half a day
[16:59] pieterh that adds effectively nothing as you invoke the callback
[17:00] nadime (a) i thought inproc was tcp on windows, so higher latency, and now i know it's not (my fault), but (b) i don't think your documentation is clear enough on support for multithreading
[17:01] Guthur something I have been meaning to ask...
[17:01] pieterh possibly
[17:01] Guthur does subscribe support regexp
[17:01] pieterh Guthur: nope, just prefix match
[17:01] Guthur ok cheers
[17:02] pieterh nadime: if you can point to anything more I can add on multithreading, to the guide, tell me
[17:02] pieterh it's hard to know what people expect, coming from other paradigms
[17:02] cremes pieterh: add a chapter on it; single line "Don't!" :)
[17:02] nadime ok, i will read through it and email the list
[17:03] pieterh cremes: half of the Guide examples are multithreaded
[17:03] cremes yes, but one socket per thread
[17:03] pieterh nadime: in very brief: use inproc from frontend to worker threads, one TCP socket to talk to the outside world
[17:03] pieterh this is a classic server model, scalable and simple
[17:04] pieterh thanks for explaining your use case, sorry if I sounded unpolite earlier
[17:05] nadime it's ok, i don't take offense, i know how annoying it is to field questions
[17:05] nadime thanks for spending so much time in here
[17:07] pieterh it's the only way to know what people are doing with 0MQ
[17:08] pieterh please do tell me how to improve the explanations of multithreading, and if you get your architecture working, it'd be cool to know
[17:08] nadime oh the architecture's been working for awhile, before i decided to launch a zillion threads on the client side ;)
[17:09] pieterh yeah, you do want to keep two cores free for your frontend
[17:09] pieterh one for the 0MQ I/O thread, one for the app thread
[17:11] nadime i've been using 4 i/o threads, but i haven't played that parameter much yet
[17:11] pieterh 4 is way too high
[17:11] pieterh inproc does not need an i/o thread
[17:11] pieterh you're not doing other I/O except the sub socket, per your explanation
[17:12] pieterh so you want precisely 1 IO thread
[17:12] nadime yep, but it's a lot of messages
[17:12] pieterh lol
[17:12] pieterh you think two threads will magically make that faster?
[17:12] nadime not magically, no :)
[17:12] nadime depends on what you're doing in your library
[17:12] nadime which i haven't read much
[17:13] pieterh well, you can set-up two or four SUB sockets, partition your data into four streams, and then use 4 IO threads
[17:13] pieterh when you have 12-core client boxes, a nice architecture
[17:13] nadime in other words: you're doing little enough processing that you would set up 1 i/o thread per i/o socket
[17:13] nadime and yes, my next plan was to add additional outward facing sockets as necessary
[17:14] pieterh in other words...
[17:14] nadime sorry, by i/o socket i mean outward facing
[17:14] pieterh 0MQ will spread sockets between I/O threads (at socket creation time!) though you can be explicit about it too
[17:15] nadime yes, i saw the affinity setup
[17:15] pieterh and you want to eliminate contention for cores
[17:15] pieterh the IO threads option should never have been placed into zmq_init()
[17:15] pieterh one more 'feature' people use without knowing why
[17:15] pieterh the only use case for it is when your main IO thread is clearly overloaded
[17:16] pieterh i.e. you're doing >80% CPU on it during peaks
[17:16] pieterh which will hit at 6-8M msg/sec or so, depending
[17:17] pieterh I'm not an expert in this area, sustrik will be able to give more precise figures
[17:18] nadime why didn't you guys dynamically create i/o threads when new non-inproc sockets are created?
[17:18] pieterh creating too many threads is as bad as creating too few
[17:18] pieterh and much easier
[17:19] nadime no, what i mean is, start with one thread per socket (i.e. don't even bother telling the user he can change the number) and then increment as you make new sockets
[17:19] pieterh yes, I understand what you mean
[17:19] pieterh how many threads per socket would you like? 1? 0.1? 0.000001?
[17:20] nadime obviously if you want it to be asynchronous you need 1
[17:20] guido_g i'd start w/ pinning the nic intr to a fixed core
[17:20] guido_g did make a difference in some tests i made
[17:20] pieterh so you have 10 sockets, of which one is a high volume SUB receiver, and 9 are low volume req/rep for other work
[17:21] nadime ah, i see your point
[17:21] pieterh your 6-core box spends its time switching between threads
[17:21] pieterh and your SUB socket mysteriously gets latency spikes
[17:21] pieterh wow, every time I do a REQ/REP, I lose 50usecs!
[17:22] pieterh whereas a single IO thread can handle all 10 sockets without even sweating
[17:22] pieterh guido_g: indeed, you want the NIC to talk to one core
[17:23] nadime you could try some dynamic management, but i guess the user knows more about his traffic than zeromq can and that's a whole big set of code to write to do intelligent prediction
[17:24] pieterh indeed
[17:26] pieterh nadime: if you want really high capacity, you'd use multiple NICs, a matching number of IO threads and SUB sockets, and multiple PUB sockets, one per segment
[17:26] nadime if you want really high capacity, you will make your own hardware
[17:26] nadime which is what the best high frequency trading shops are doing
[17:27] pieterh 'make' = 'have built to custom specs'?
[17:27] pieterh or literally going out and boiling sand down into CPUs...?
[17:28] nadime some are actually doing their own hardware creation, i.e. having someone build chips for them
[17:28] pieterh FPGAs...
[17:28] nadime some are using FPGAs
[17:28] nadime some just use custom NICs with the ability to write assembly onto the nic
[17:29] nadime it's an expensive business, which is why i am not in it!
[17:29] pieterh it'll all end in tears as they fight for socket space on the exchange motherboard
[17:30] nadime it will all end in tears when it's regulated away, in any case
[17:31] pieterh you're really doing 4M msg/sec on Windows?
[17:31] nadime yep
[17:31] pieterh I think that's a world record
[17:31] nadime small msgs
[17:31] nadime it's more like 3.6
[17:31] nadime and that's peak
[17:32] pieterh still extraordinary
[17:32] pieterh nice
[17:32] nadime it's very cool, let's us normalize our data feeds between our data providers so that our quants don't have to code to specific data provider specs, but still run historical backtests at many times realtime
[17:34] pieterh clearly
[17:34] cremes pieterh: new topic... did you ever have any luck reproducing issue #174?
[17:35] pieterh cremes: not yet... I was looking at it yesterday
[17:35] pieterh we do want to resolve that for 2.1 stable
[17:35] cremes great; anything i can do to help, let me know
[17:36] pieterh will do
[17:36] cremes i've worked around it by using inproc (my new favorite transport)
[17:36] pieterh :-) inproc is nice
[17:36] cremes we really need to rename it for 3.0 though
[17:36] nadime why not call it local?
[17:37] cremes nadime: it only works between sockets in the *same* context
[17:37] cremes so :incontext or similar would be more accurate
[17:37] nadime is there a reason to create more than one context per application that i am unaware of?
[17:37] cremes inproc makes it sound as though it can be used within the same process regardless of other restrictions
[17:38] cremes nadime: when interfacing to a 3rd party library that also uses 0mq, for example
[17:38] nadime yeah i guess if they make their context private
[17:39] nadime our they don't match what you need on the i/o thread front
[17:39] cremes 99% of the time you only need 1 context per process
[17:39] cremes the fear is that you could have endpoint collisions, e.g. same name for an ipc or inproc transport
[17:39] cremes also, the default max_sockets for 0mq is 512; if you don't know how many the lib is using you could run out
[17:40] nadime right
[17:40] cremes so it comes down to properly segregating your resources
[17:40] pieterh cremes: what's wrong with the old suggestion itc:
[17:40] pieterh or simply ctx:
[17:40] cremes pieterh: what did that stand for again? inter-thread-communication or similar?
[17:40] pieterh yeah
[17:40] cremes i like :intracontext
[17:41] cremes it says everything (to me)
[17:41] pieterh why not usethesamecontextifyouexpectthistowork:// ?
[17:41] cremes best one yet!
[17:42] pieterh nadime: there are quite a lot of cases where processes use multiple contexts
[17:42] pieterh e.g. you could wrap the SUB socket up in a framework that creates/destroys a context automatically
[17:43] nadime that's what i do, but i use a factory
[17:43] nadime as in: if context exists, return, otherwise create one
[17:43] pieterh I mean, you can create a context invisibly, use that for just one object, terminate it when you destroy the object
[17:44] pieterh as long as that socket works over TCP, not inproc
[18:23] private_meta pieterh: do you want the version assert in the later c++ examples as well?
[18:34] Bob is there any way to find out when a connection has been made to a PUB socket?
[18:35] Bob I have two applications that are subscribing to each other's pub sockets to transfer messages between them, but I'm loosing messages because connection order is non-deterministic
[18:36] guido_g no
[18:36] Bob is Pub/Sub the right model to use for that?
[18:37] Bob here's what I'm trying to do: I have a number of simulation engines that are started up and share messages between them
[18:37] guido_g you've to describe the use case
[18:38] Bob I have a multicast beacon that each sim engine broadcasts to identify itself. When another engine receives the beacon message (which contains it's pub socket address/port) it connects to it if hasn't already
[18:38] Bob over time this gives me a fully connected network of sim engines (which is what I think I want)
[18:39] Bob but I need to send some startup messages on connect to make the engine forward certain types of messages over it's network connection
[18:39] guido_g why not using one multicast channel for that?
[18:39] Bob guido_g: I'm not sure I follow what you mean. I am using a multicast channel for identifying sim engines
[18:39] guido_g btw, pub/sub doesn't do connections, it's more like a radion transmission
[18:40] guido_g s/radion/radio/
[18:40] Bob ok
[18:40] guido_g just have bus structure for all participants, will make alles messages available to all subscribed nodes
[18:48] Bob guido_g: thanks! I guess I need to rethink my connection scheme. maybe creating new pair sockets is the way to go.
[18:48] guido_g no, pair ist not the way to go
[18:49] Bob you think pub/sub is right?
[18:50] guido_g if you want to communicate from every node to every other node a bus structure would fit
[18:59] private_meta pieterh: btw, building single files doesn't seem to be possible with the C++ builder
[19:10] private_meta pieterh: interestingly, if i simulate the behavior in the C build file, it still doesn't work
[19:16] private_meta ah ok, fixed
[19:33] private_meta pieterh: maybe I made a mistake, but spqueue.c, as found in the repo, seems broken
[19:34] private_meta pieterh: it never let's the client connect. I haven't yet had the time to look properly though, I'll check again tomorrow
[19:58] mikko pieterh: it's ready now
[19:58] mikko tested quickly locally
[20:22] pieterh private_meta: version assert... normally only if the code needs 2.1+ functionality
[20:23] pieterh mikko: yay!
[20:23] mikko pieterh: daily builds around 5am should run the same thing on more platforms
[20:23] mikko im exchausted
[20:26] pieterh mikko: can you tell me very briefly what the new tooling fixes?
[20:26] pieterh openpgm 5.1.113, obviously
[20:26] pieterh builds on specific platforms?
[20:26] mikko updated openpgm, removes the build duplication from zeromq side
[20:27] pieterh so it uses the openpgm build directly, right
[20:27] mikko the biggest benefit is that now it should be easier to upgrade openpgm in the future
[20:27] mikko yes
[20:27] mikko it invokes openpgm autotools build
[20:27] mikko the 'old way' was a huge hack
[20:28] pieterh this is wonderful
[20:28] mikko also, there is a hidden feature
[20:28] mikko it's not advertised much as its not well tested
[20:28] mikko you can drop in new version of libpgm
[20:28] mikko and do:
[20:28] mikko ./configure --with-pgm=libpgm-x.y.z
[20:29] mikko it searches for libpgm-x.y.z.tar.gz in foreign/openpgm
[20:29] pieterh ah, you mentioned that... sweet
[20:29] mikko also, autotools builds on both packages paves way for including them in distros
[20:29] mikko which would be ideal in longer run
[20:29] mikko hopefully the new build doesn't break many things
[20:32] pieterh mikko: I'll commit this to the 2.1 and push, then we can test it with the autobuild system...
[20:33] mikko pieterh: yeah
[20:33] mikko sounds good
[20:33] pieterh done
[20:33] pieterh excellent, lovely, amazing stuff!
[20:33] mikko we can always revert if things break left right and center
[20:33] pieterh nah, never back, always forwards!
[20:33] pieterh after that we can prepare a patch to send to Sustrik upstream...
[20:35] pieterh g'nite, my kids are destroying my TV, I need to intervene...
[20:37] mikko nite!
[20:59] Guthur pieterh, how many kids you got, if you don't mind me asking?
[21:30] private_meta pieterh: I figured that, but zhelpers in c++ doesn't offer the functionality, so it's missing in all files up to now
[22:16] guido_g good night all!
[22:17] Guthur night guido_g

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy