IRC Log


Wednesday October 13, 2010

[Time] NameMessage
[02:02] c00w Can anyone help me with installing zeromq? I keep getting errors when I try to compile files.
[02:07] kylezoa and what exactly are those errors?
[02:08] c00w I think I have it installed incorrectly. It compiles but the linker throws stuff like /tmp/ccW4H1WP.o: In function `zmq::error_t::error_t()': server.cpp:(.text._ZN3zmq7error_tC1Ev[zmq::error_t::error_t()]+0x27): undefined reference to `zmq_errno''
[02:08] kylezoa i'm not familiar with working with C bindings sorry
[02:09] c00w c++ binding...
[02:09] kylezoa or that
[02:09] c00w k.
[02:11] c00w I figured it out.
[02:16] delaney i sent a question to the mailing list, how long does it usually take to go through?
[03:59] delaney hey andrewvc
[03:59] andrewvc yo
[06:00] CIA-14 zeromq2: 03Sebastian Otaegui 07maint * r2568947 10/ (AUTHORS builds/redhat/zeromq.spec):
[06:00] CIA-14 zeromq2: Add support for RHEL6 in the spec file
[06:00] CIA-14 zeromq2: Signed-off-by: Sebastian Otaegui <feniix@gmail.com> - http://bit.ly/cKPxfL
[06:01] CIA-14 zeromq2: 03Sebastian Otaegui 07master * r2568947 10/ (AUTHORS builds/redhat/zeromq.spec):
[06:01] CIA-14 zeromq2: Add support for RHEL6 in the spec file
[06:01] CIA-14 zeromq2: Signed-off-by: Sebastian Otaegui <feniix@gmail.com> - http://bit.ly/cKPxfL
[06:01] CIA-14 zeromq2: 03Martin Sustrik 07master * r5ae878b 10/ (AUTHORS builds/redhat/zeromq.spec):
[06:01] CIA-14 zeromq2: Merge branch 'maint'
[06:01] CIA-14 zeromq2: * maint:
[06:01] CIA-14 zeromq2: Add support for RHEL6 in the spec file - http://bit.ly/9PBW4n
[08:12] CIA-14 zeromq2: 03Martin Sustrik 07master * r9d96e00 10/ (8 files):
[08:12] CIA-14 zeromq2: Clean-up of the code related to attaching/detaching engines to sessions.
[08:12] CIA-14 zeromq2: Session base class now handles the engine events exclusively. It notifies
[08:12] CIA-14 zeromq2: derived session types using dedicated "attached" and "detached" events.
[08:12] CIA-14 zeromq2: Couple of bugs was fixed along the way.
[08:12] CIA-14 zeromq2: Signed-off-by: Martin Sustrik <sustrik@250bpm.com> - http://bit.ly/bPyigb
[09:03] phaserescu hi! is it possible to use zeromq on iphone? has anyone tried it?
[09:04] mikko phaserescu: haven't tried but it would be interesting to test
[09:32] xraid hi, who maintains http://www.zeromq.org/docs:bindings
[09:33] xraid there is a small typo on socket part where a paragraph is before the socket heading ...
[09:33] mikko xraid: pieterh as far as i know
[09:34] mikko xraid: but anyone with sufficient rights should be able to fix errors
[09:37] sustrik xraid: it's wiki, feel fee to fix it
[09:38] xraid k - done ;-)
[09:38] mikko sustrik: i thought you need access
[09:39] sustrik no, you just need to create an account
[09:39] sustrik you've used to need access ages ago
[09:39] sustrik but since we've moved to the community ownership, anyone is free to edit
[09:51] xraid also something is up with the helper c in zguide c++
[09:52] xraid that with a -p flag = c++ and .oo it creates .o and fails
[09:52] sustrik xraid: report it on the mailing list presumably
[09:59] pieterh xraid: hang on, I'm looking at something else
[09:59] pieterh will be back in 2 mins
[10:00] pieterh sustrik: can we have as design goal that 0MQ sockets do not die when given invalid messages?
[10:01] pieterh this seems a perverse use of asserts
[10:04] pieterh xraid: re
[10:04] pieterh when you say 'it fails' what do you mean exactly?
[10:05] pieterh sustrik: specifically, when reading invalid messages off the network
[10:05] xraid well it ait bulding since its supposed to se .oo but generated .o
[10:05] pieterh it's reasonable that APIs assert when called incorrectly
[10:05] pieterh but it's pathological that services assert when they get invalid data off the wire
[10:06] pieterh xraid: where is it looking for an .oo?
[10:06] pieterh let me try... hang on
[10:06] xraid pieterh_: try it ... when invoking c helper
[10:06] xraid with -p flag
[10:07] pieterh 'c -p lruqueue' works fine
[10:07] pieterh what are you trying that does not work?
[10:07] pieterh specifics, please :-)
[10:08] xraid hmmm try ./build
[10:08] pieterh './build all'?
[10:08] pieterh sorry, I don't like guessing
[10:08] xraid it when linking it fails = -lzmq
[10:09] pieterh i just ran './build all' and it worked fine
[10:09] xraid it builds a .o linking looks for .oo
[10:09] xraid ahhh
[10:09] pieterh please tell me what directory you are in, and exactly what commands you are trying
[10:09] pieterh i cannot guess this
[10:13] xraid C++ i get i686-apple-darwin9-g++-4.2.1: lruqueue.opp: No such file or directory in lruqueue.lst
[10:13] pieterh xraid: you are in examples/C++?
[10:13] pieterh what command are you typing, exactly?
[10:14] xraid ./build all or ./c -p lruqueue
[10:14] pieterh ok...
[10:15] xraid so it builds a lruqueue.o
[10:15] xraid but no lruqueue.opp
[10:15] xraid taht linking wants ?
[10:15] pieterh try './c -v -p lruqueue'
[10:16] pieterh tell me what it prints
[10:16] sustrik pieterh_: depends on what you mean by invalid messages
[10:16] sustrik it they are straight corrupted, 0mq will fail
[10:16] xraid Compiling lruqueue (g++ -c -O2 -D_REENTRANT -D_GNU_SOURCE -Wall -Wno-unused -fno-strict-aliasing -DBASE_THREADSAFE -I -o lruqueue.opp lruqueue.cpp)...
[10:16] xraid i686-apple-darwin9-g++-4.2.1: lruqueue.opp: No such file or directory
[10:16] pieterh sustrik: I mean, for example, req.cpp asserting when not getting a multipart message
[10:16] sustrik as it will try to access invalid memory etc.
[10:17] sustrik sure, that should not happen
[10:17] pieterh sustrik: in any case, a service should always be robust against invalid data
[10:17] pieterh xraid: so, that command looks correct
[10:17] pieterh '-o lruqueue.opp'...
[10:17] sustrik there are some invalid data you cannot be robust against
[10:18] sustrik at least in user space
[10:18] pieterh that is debatable
[10:18] pieterh user space, sure
[10:18] pieterh but not wire data
[10:18] xraid i got a generated lruqueue.o
[10:18] sustrik sure
[10:18] pieterh sustrik: I'm talking only and specifically about wire data
[10:19] pieterh we've had this issue of asserts for a long time in 0MQ
[10:19] sustrik yes, report the problem and i'll fix it
[10:19] pieterh I'd like a policy statement about not crashing / asserting when passed invalid wire data
[10:19] sustrik "it should not happen"
[10:19] pieterh at least so people know this is a bug to report and not a deliberate design decision
[10:19] sustrik it does but it should not
[10:20] pieterh :-)
[10:21] sustrik btw, mato promised to do an audit of the code to make it safe wrt what comes on the wire
[10:23] sustrik and we need a policy to deal with poisoned connections
[10:24] pieterh poisoned = ?
[10:24] sustrik sending invalid data
[10:24] pieterh ah, ok: report and disconnect and if we want to get sophisticated, blacklist
[10:24] pieterh but usually blacklisting can be done elsewhere if we report IP addresses
[10:25] sustrik drop old messages from this connection?
[10:25] pieterh Well, if this happens today it's 100% likely to be a programming error
[10:25] sustrik drop the whole identity (long-lived session) or just a TCP connection?
[10:25] sustrik etc.
[10:25] pieterh so the connection has to be killed, loudly reported, and treated as 'broken'
[10:25] pieterh drop everything, I'd say
[10:25] pieterh you can't trust it, if it sends invalid data
[10:26] pieterh like in AMQP we had connection level assertions
[10:26] pieterh any error = kill connection
[10:26] pieterh that was very robust
[10:26] sustrik connection = long-lived session?
[10:26] sustrik or TCP connection?
[10:27] sustrik anyway, i don't know what the right policy is myself
[10:27] pieterh in 0MQ's case, I'd say the session
[10:27] pieterh clearly, since a session maps to an application
[10:27] pieterh and the problem is with the application
[10:27] sustrik to be discussed
[10:27] pieterh this is really an implementation detail
[10:27] pieterh it's barely relevant
[10:27] sustrik specification detail rather
[10:27] pieterh the only important aspects are:
[10:27] pieterh 1. do not crash / assert
[10:28] pieterh 2. report explicitly in a way that cannot be overlooked by accident
[10:28] sustrik yup, mato is up to that
[10:28] pieterh that's all that matters
[10:28] pieterh at least today, because we don't have more knowledge of use cases
[10:28] pieterh opinion is worthless
[10:29] pieterh as to testing this, it's fairly simple IMO, make it a challenge for the community
[10:30] pieterh anyone who can crash a 0MQ socket with wire data gets one point
[10:30] pieterh (first report only)
[10:30] pieterh highest score gets free trip to first 0MQ conference
[10:32] sustrik easy way to break it: send a message header stating that message size is 358 TB
[10:32] sustrik do i get a point?
[10:32] pieterh so, you score 1 point
[10:32] sustrik :)
[10:33] pieterh also, anyone proposing a patch to fix a reported crash gets 1 point
[10:33] pieterh game on?
[10:33] xraid pieterh_: so c don't link in zguide/examples/C++ -- only generates a .o file and fails creating a binary executable ...
[10:34] xraid dont do TB in message size ;-)
[10:34] pieterh xraid, I've no idea why gcc is not working properly on your system
[10:34] xraid k its local you say ?
[10:34] pieterh the command line syntax seems clear and correct, yes
[10:34] pieterh you can try it by hand
[10:49] xraid so './c -p -l -lzmq durapub.cpp' works for you ?
[10:49] xraid it might be the Darwin g++ doing the .opp thing ?
[10:50] xraid i get durapub.opp: No such file or directory
[10:53] sustrik pieterh_: beware of mato winning the competition!
[10:54] sustrik he may be at new zealand by then :)
[10:56] mikko pieterh_: does assert failure count as crash?
[10:56] mikko lestrrat: there?
[11:06] pieterh sustrik: np
[11:06] pieterh mikko: yes
[11:07] pieterh xraid: IMO it's the Darwin g++ doing something weird
[11:07] pieterh check the man pages for the -o switch and try it by hand
[11:14] xraid so you guys on windoze ? ;-) or use linux desktop ?
[11:27] caljunior I'm a recent convert to "the way of the zmq" and had success using the jzmq binding. I am currently trying to get Excel to join the zmq party.
[11:29] caljunior I understand there is no VB binding currently available. Could someone point me in the right direction. Thanks.
[11:29] pieterh xraid: mostly on Linux
[11:29] pieterh caljunior: hmm, there is a Basic bindings project, it might be a start
[11:32] caljunior pieterh_ yes that is a start. do you know of anyone who tried this before and shared his/her experiences on the web?
[11:34] pieterh caljunior: nope
[11:34] pieterh sorry
[11:34] pieterh but can't you use C# from Excel?
[11:40] caljunior A deprecated white paper discussed a .net extension. I'll give it a try. (http://www.zeromq.org/whitepapers:design-v05/)
[11:41] jsimmons wait by VB you mean VB6?
[11:42] pieterh caljunior: there is a .net/c# binding
[11:43] caljunior jsimmons: actually I mean VBA 7.0
[12:00] caljunior pieterh_ thanks. also for 0mq. it opens the mind, not just the software. :-)
[12:05] pieterh :-)
[12:06] sustrik caljunior: isn't Excel accessible from .NET?
[12:07] caljunior yes it is.
[12:07] sustrik so does .NET binding suit your needs?
[12:08] sustrik interacts with .net
[12:11] caljunior I'm a unix/java/clojure guy trying my hand at Excel programming so I am a bit out of my depth here. But it needn't be VBA. All I am trying to accomplish is setup a publish subscribe system between jzmq and excel.
[12:13] sustrik caljunior: i am not an expert either
[12:14] sustrik however, if you succeed your findings would be extremely valuable
[12:14] sustrik so please, do share the on the mailing list or on the wiki
[12:15] caljunior VBA was my first thought. I quick google search tells be VB.net is another option.
[12:15] caljunior I will share what I learn.
[12:16] sustrik thanks, lot of people would benefit from that
[12:16] mikko caljunior: you need applescript version for ms office for mac as well :)
[12:18] caljunior Am I being ridiculed here? :-)
[12:19] mikko no, just saying
[12:19] mikko i use excel a lot at work and noticed that VB doesn't work on mac
[12:19] caljunior I believe Excel for Mac 2011 will have VB support.
[12:21] sustrik in theory mono should run on osx
[12:22] mikko sustrik: but the excel itself doesn't support it (yet)
[12:22] sustrik doesn't support .net or doesn't support mono?
[12:22] caljunior mikko: it probably never will
[12:23] caljunior sustrik: excel for mac doesn't support mono. .net doesn't support osx.
[12:23] mikko sustrik: neither (on mac)
[12:24] keffo mono does run on osx
[12:24] mikko keffo: yes
[12:25] sustrik :\
[12:27] pieterh caljunior: so what interface languages does Excel on Mac support?
[12:27] mikko pieterh_: applescript iirc
[12:27] caljunior pieterh: essentially none. you can do some very basic stuff using applescript
[12:28] caljunior Excel for Mac is a joke.
[12:28] pieterh well, what are your data rates?
[12:28] caljunior data rates?
[12:28] pieterh there are many ways to skin a bear
[12:29] pieterh how many messages per second do you need to push in or out of your spreadsheet?
[12:29] pieterh a few updates per second?
[12:30] caljunior more like 1800 per second
[12:30] keffo hehe
[12:30] mikko from a spreadsheet?
[12:31] pieterh i assume you process these in some way, and display the results...?
[12:31] pieterh you can't literally update 1,800 cells per second, can you?
[12:31] caljunior I try to keep the processing out of excel
[12:31] caljunior in jvm
[12:31] pieterh ok, right
[12:31] pieterh so imagine you have a subscriber that does the heavy work in Java
[12:32] pieterh and publishes its own results via some other route that Applescript can easily get
[12:32] pieterh not 0MQ
[12:32] mikko what is the application doing, may i ask?
[12:32] caljunior well the work is done in jvm but publishes to excel for monitoring.
[12:32] caljunior mikko: trading
[12:33] pieterh from my close-to-nil knowledge of applescript I'd guess you want to send AppleEvents from your Java code to Excel
[12:33] caljunior pieterh_ yes
[12:34] pieterh could be quite nice but not at all portable
[12:34] pieterh anyhow, forget putting 0MQ into Excel on Mac
[12:34] mikko may i ask, why excel?
[12:34] pieterh mikko: excel is the tool of choice for traders who want to see visuals
[12:34] caljunior no. not a problem it is not meant for distribution. only in-house.
[12:34] pieterh lets you make massive errors with just a single click
[12:35] mikko one would imagine you could easily make a gui that would contain the visuals they need
[12:35] mikko either web-based or desktop app
[12:36] pieterh mikko: IME it's the programmability Excel gives that people love
[12:36] caljunior excel is a malleable gui.
[12:36] pieterh doesn't require IT support to do fancy stuff
[12:36] sustrik actuall, excel Is used in trading environment quite a lot afaik
[12:36] mikko thats rather interesting
[12:36] pieterh caljunior: so you do 0MQ pubsub to your Java code, and then AppleEvents to push to Excel
[12:36] sustrik heard of RTD?
[12:37] mikko sustrik: realtime trading?
[12:37] pieterh if anyone wants to make a smooth integration of 0MQ and Excel they could make a lot of money
[12:37] sustrik real-time access to excel
[12:37] sustrik strange but true
[12:37] pieterh like lawyers all using WP
[12:37] keffo that sounds easy enough?
[12:38] pieterh keffo: probably, yes
[12:39] keffo too bad everything related to excel makes me feel dirty :)
[12:40] sustrik better take a shower
[12:40] sustrik it may be contagious, who knows
[12:41] sustrik in any case, the use case is not a fake, it's real
[12:41] sustrik that's why i'm saying the feedback would be valuable
[12:42] sustrik others may be solving the same problem
[12:42] mikko the excel side looks pretty straight-forward
[12:42] caljunior I'm sure they are.
[12:42] mikko would be interesting to have PUB socket that publishes like "E1|value_here"
[12:42] mikko etc
[12:55] caljunior pieterh_ the complication is that the java code runs on a server. that's where I would like zeromq to provide the link to excel.
[12:56] caljunior AppleEvents won't do that for me.
[12:56] mikko caljunior: do you have a lot of mac users?
[12:57] caljunior no
[12:58] caljunior But whether it's mac or windows is not the issue.
[12:58] mikko for windows C#.NET looks doable
[12:59] mikko based on http://support.microsoft.com/kb/302084
[13:01] caljunior That's a useful link.
[13:01] mikko http://www.zeromq.org/bindings:clr
[13:02] mikko you should be able to use that binding together with the C#.NET program
[13:07] pieterh caljunior: you'll need to run a bridge process on each Mac IMO
[13:09] caljunior bridge process?
[13:09] pieterh "C# is the microsoft spawn of satan java hack that is windows only. Go C++ or use the OSX development tools from Apple"
[13:10] pieterh so you want to connect Excel on Mac to random 0MQ network
[13:10] pieterh you can't use C#, can't use VBA
[13:10] caljunior yes
[13:10] pieterh you can only use AppleEvents
[13:10] pieterh so you have a small proxy / bridge that runs locally
[13:10] caljunior right
[13:10] pieterh it uses a 0MQ SUB socket to pull data from a central publisher
[13:11] pieterh it then forwards that using AppleEvents
[13:11] caljunior got it
[13:11] pieterh you could write it in any language
[13:11] pieterh feel free to open source the results so others can improve / benefit from it
[13:11] mikko pieterh_: nope
[13:11] pieterh this bridge technique is the standard answer to "how do I connect (random weird system) to 0MQ"
[13:11] pieterh mikko: ?
[13:11] mikko you can allow remote appleevents
[13:12] mikko in System Preferences -> Sharing
[13:12] mikko http://docs.info.apple.com/article.html?path=mac/10.4/en/mh896.html
[13:13] pieterh however, do you then have routing to a whole set of workstations?
[13:13] pieterh you need a pubsub framework to distribute events from 1 pub to N subs
[13:13] mikko not sure if you can multicast them
[13:13] pieterh then it won't work
[13:13] pieterh or be way more complex
[13:14] mikko 3031 TCP/UDP Remote AppleEvents
[13:14] caljunior * dreaming of a zeromq add-in for excel
[13:14] mikko but yes, bridge process might be easier
[13:14] pieterh bridge process is KISS
[13:15] mikko sub process that publishes messages as apple events
[13:15] caljunior I like KISS
[13:15] pieterh yes
[13:15] keffo kiss rules
[13:16] pieterh caljunior, you can probably start/stop the bridge from within the Excel app
[13:17] keffo caljunior, What is the
[13:17] keffo ...
[13:17] keffo ..usecase for this?
[13:18] caljunior getting real time data into excel and rpc
[13:19] keffo like for instance stock value or something like that?
[13:19] keffo pushed, that is?
[13:19] caljunior yep
[13:19] caljunior server push
[13:20] keffo and where is your shortcoming, ie, what parts do you miss?
[13:20] keffo everything? :)
[13:20] caljunior the server side is clear
[13:20] caljunior using excel as a client through zeromq is what this discussion is about
[13:20] keffo ok
[13:21] keffo and excel can only be extended through which language?
[13:21] caljunior depends on the platform
[13:21] keffo c++? =)
[13:22] caljunior I would like to have both Windows and Mac supported. Mac will get Excel VB support in Dec-10
[13:22] caljunior So my first idea was VBA.
[13:23] keffo VB is the answer to a question that will never be asked.
[13:23] caljunior C# binding is available but only on Windows of course.
[13:23] keffo how come, shouldnt mono work out of the box, being abi compatible?
[13:23] pieterh keffo: Excel on Mac apparently only speaks AppleEvents
[13:24] keffo or, cli comp. is maybe a better word
[13:24] keffo ah
[13:24] caljunior http://lists.ximian.com/pipermail/mono-osx/2006-September/000603.html
[13:25] keffo what about ExcelGrid? =)
[13:27] caljunior sustrik: clrzmq seems to be out of date: version 2.0.7 on github.
[13:29] caljunior keffo: as you can read above we came to the conclusion that a local bridge would be the solution.
[13:29] keffo caljunior, Couldnt you open a pipe from vba, and just write to a zmq-aware daemon?
[13:30] keffo and you wanted 1800 msgs/s, you said?
[13:30] keffo that's a fair amount
[13:33] caljunior keffo: zmq-aware daemon sounds like a bridge to me
[13:35] keffo indeed :)
[13:35] keffo It just bypassed the need for appletalk, which also makes me feel dirty :)
[13:35] keffo or, applevents!
[13:36] sustrik caljunior: what's the problem you are seeing?
[13:37] caljunior none so far. just checking if there have been major changes that I would run into.
[13:37] caljunior I will post any bugs.
[13:38] sustrik caljunior: where have you got the version number from?
[13:47] mikko can one compile clrzmq on linux?
[13:47] mikko i could add to hudson if it's possible
[13:47] sustrik in theory, with mono
[13:47] sustrik i wouldn't even try :)
[13:48] mikko sounds like a challenge
[13:50] keffo it should build fine as far as I can tell?
[13:51] mikko Done building project "/tmp/clrzmq/clrzmq.sln".-- FAILED
[13:51] mikko /usr/lib/mono/3.5/Microsoft.Common.targets: warning : Reference 'System.Data.DataSetExtensions' not resolved
[13:52] mikko just a warning
[13:52] mikko hmm
[13:53] keffo hmm indeed
[13:53] sustrik no idea, guys
[13:54] keffo why is that there?
[13:54] mikko : error CS0040: Unexpected debug information initialization error `The assembly for default symbol writer cannot be loaded'
[13:54] sustrik it would be good to have someone who really understands the stuff...
[13:54] mikko installing monodevelop
[13:54] mikko it _will_ build on hudson
[13:54] sustrik sounds like something with your installation though...
[13:59] mikko Build succeeded. 0 Warning(s) 0 Error(s)
[13:59] mikko adding to hudson now
[14:00] sustrik wow
[14:00] sustrik nobody have built that stuff in years
[14:01] keffo .../mounting /dev on /root/dev failed: No such file or directory
[14:01] keffo score!
[14:01] mikko # mono local_lat/bin/Debug/local_lat.exe
[14:01] mikko usage: local_lat <address> <message-size> <roundtrip-count>
[14:02] sustrik :)
[14:02] mikko Unhandled Exception: System.DllNotFoundException: libzmq
[14:02] mikko dies on that if i try to use it
[14:02] mikko (expected)
[14:02] pieterh path to dll... imo
[14:02] mikko yep
[14:03] mikko haven't got mono build of libzmq (yet)
[14:03] mikko errmmm
[14:03] keffo you need to tell mono which .so to use as dll
[14:04] keffo ie, dllmap
[14:05] keffo <dllmap dll="somelib" target="thelib.so"/>
[14:06] keffo or, remove the ".dll" in the dllimport attribute in zmq.cs :)
[14:09] mikko how do i load dllmap?
[14:09] mikko mono --config ?
[14:09] keffo app.config
[14:09] keffo w8
[14:09] mikko actually, using LD_LIBRARY_PATH should work
[14:09] keffo www.mono-project.com/Config_DllMap
[14:10] mikko yep
[14:10] mikko runs
[14:11] keffo where did you use ld_library_path?
[14:11] mikko LD_LIBRARY_PATH=/path mono
[14:11] mikko because the dll name matches the .so name
[14:11] mikko so it automatically tries to load "libzmq.so"
[14:12] keffo nice
[14:12] mikko https://gist.github.com/f5a0d4847258db4bb7b8
[14:14] mikko http://valokuva.org:8080/job/clrzmq_maint/1/console
[14:17] keffo I wonder why mono has no plans to implement wpf..
[15:08] mikko sustrik: ** (remote_lat/bin/Debug/remote_lat.exe:27949): WARNING **: Shutting down finalizer thread timed out.
[15:08] mikko against master branch (clrzmq)
[15:08] mikko anyways, master and maint builds added to hudson
[15:09] mikko should build after relevant 0MQ master and maint have been buil
[15:09] mikko t
[16:09] sustrik mikko: was there local_lat running?
[16:10] sustrik mato: you there?
[16:11] mato sustrik: yes
[16:11] sustrik what's the _MSC_VER stuff?
[16:11] mato ?
[16:11] mato what what?
[16:11] sustrik does it makes sense to use __rdtsc() with mingw?
[16:11] mato I have no idea, someone needs to try it and see
[16:12] mato I'd expect it to work though
[16:12] sustrik it's an intrinsic function afaik
[16:12] sustrik = emulated by compiler
[16:12] sustrik let me check...
[16:12] mato as I said, I have no idea if MinGW implements those in order to be compatible with MSVC
[16:12] mato ask Steve McCoy maybe, he might know.
[16:13] sustrik in any case, what's wrong with unsing _MSC_VER?
[16:13] mato If it doesn't then obviously _MSC_VER needs to be used, but then some alternative needs to be provided or at least #error
[16:13] mato Well, *something* needs to happen on Win32 and not on MSVC
[16:14] sustrik it fails over to #ifdef __GNUC__
[16:14] sustrik which has a snippet of machine code there that does the same thing
[16:14] mato ah, it does, ok then
[16:15] mato in that case, yes, _MSC_VER should be used for determining if intrin.h is included
[16:15] sustrik ack
[16:23] mikko sustrik: yes
[16:47] delaney sorry to keep bothering you guys.... if i call data = socket.recv_multipart() it works fine... if i add flags=zmq.NOBLOCK then i'm getting
[16:47] delaney data = socket.recv_multipart(flags=zmq.NOBLOCK)
[16:47] delaney File "_zmq.pyx", line 883, in zmq._zmq.Socket.recv_multipart (zmq\_zmq.c:6802)
[16:47] delaney File "_zmq.pyx", line 806, in zmq._zmq.Socket.recv (zmq\_zmq.c:5997)
[16:47] delaney File "_zmq.pyx", line 836, in zmq._zmq.Socket._recv_copy (zmq\_zmq.c:6368)
[16:47] delaney zmq._zmq.ZMQError: Resource temporarily unavailable
[16:53] sustrik it means there's no message available right now
[16:53] sustrik EAGAIN
[16:53] delaney but wouldn't/shouldn't pyzmq catch that?
[16:54] delaney it says in the zmq/core/socket.pyx If NOBLOCK is set, this method  will return None if a message is not ready.
[16:54] delaney line 470
[16:59] delaney i'll write a ticket on pyzmq, easy fix
[17:23] pieterh sustrik: issue 94 looks solved now, thanks
[17:24] pieterh I got a new assertion failure but only once, and could not get it again, it's at http://github.com/zeromq/zeromq2/issues/issue/100
[17:30] sustrik pieterh_: ok
[19:04] kev009 is it possible to use zmq to build a server for an existing TCP protocol?
[19:05] pieterh kev009: indirectly, yes
[19:05] pieterh that is, you need to handle the existing TCP protocol yourself using old fashioned sockets
[19:06] pieterh but you can use 0MQ to bridge that to a fast multithreaded architecture for the actual server
[19:06] kev009 so zmq would only be for IPC of the server?
[19:06] pieterh inproc and ipc, yes
[19:06] pieterh it's the best way to build multithreaded apps
[19:06] pieterh and any real TCP server has to be multithreaded in one way or another
[19:07] pieterh 'real' = 'large scale'
[19:07] kev009 right. I'm looking to build a game server and have only used sockets in the past but bookmarked zmq a while back
[19:08] pieterh there is already a network of game clients?
[19:08] kev009 yes, minecraft
[19:09] pieterh ah, well, how I would do this is write a very thin TCP-to-0MQ proxy
[19:09] pieterh single threaded
[19:09] pieterh and then use 0MQ to build the actual server architecture
[19:10] pieterh the proxy can actually be a thread in a multithreaded process
[19:12] kev009 hmm.. wouldn't each client want a TCP proxy thread so they don't block, or is that extremely quick?
[19:12] kev009 the goal will be to eventually support tons of clients
[19:13] pieterh you can do this in one thread
[19:13] pieterh the usual technique is to use a poll / epoll / select loop
[19:14] pieterh you manage a whole set of sockets (can be 10k or more sockets)
[19:14] pieterh and you process activity on any socket
[19:14] pieterh it's not ... trivial
[19:14] pieterh but it's part of any server architecture anyhow
[19:15] pieterh it depends on the complexity of the protocol as well
[19:16] pieterh but in general if you're not going any slow i/o (disk) or processing, you can handle a *lot* of client connections in pseudo-parallel from one thread
[19:16] kev009 interesting, ok
[19:16] pieterh imagine making an HTTP server like this...
[19:16] pieterh so you have a set of open client sockets
[19:17] kev009 do you know of any articles or books that cover modern daemon development (at least the epoll, proxy stuff)
[19:17] pieterh and each client socket has a context, i.e. state associated with it
[19:17] pieterh shrug... none of this is very recent art
[19:17] pieterh take a look at mongrel2
[19:18] pieterh you usually want to use a state machine to process input on each client socket
[19:19] pieterh I think mongrel2 is the best example of such an architecture I know
[19:19] kev009 ok, grabbing the source now
[19:20] pieterh zed shaw is using ragel afaik
[19:20] pieterh we used to do this in Xitami using the Libero state machine tool
[19:20] pieterh could survive a slashdotting on a simple windows box...
[19:35] mathijs Hi all, I would like to implement some kind of "transactions" on top of 0mq. A certain piece of code gathers data (versioned copy of some state) from a few places(boxes), performs some calculations, and updates the state by sending "update to XX if version still is Y" messages.
[19:36] mathijs However, I would like to be able to do this atomically. so if a transaction touches A and B (both residing on different boxes), I only want to "commit" the result, if the versions of both A _and_ B are still the same.
[19:45] pieterh hi mathijs
[19:45] mathijs hi pieter
[19:46] pieterh do you know how an XA transaction manager works?
[19:48] pieterh basically, if I remember this correctly, you'd need a transaction coordinator
[19:48] pieterh all nodes have to signal 'ready' to the coordinator
[19:48] pieterh and then they all have to wait
[19:49] pieterh the coordinator then tells them all, 'commit'
[19:49] pieterh and they all have to commit but keep the previous state ready
[19:49] pieterh then the coordinator confirms, or cancels, or disappears
[19:51] mathijs pieterh_: so what if the 'commit' signal doesn't get to one of the nodes?
[19:51] mathijs or it gets there, but a bit (few seconds) later
[19:53] pieterh if the nodes don't all confirm to the coordinator that they got the commit, it aborts the transaction
[19:53] pieterh and they all have to rollback
[19:54] pieterh i know this still appears to have a loop hole
[19:54] mathijs they all confirm they have received (not applied) the commit. then the coordinator sends "apply"
[19:55] pieterh http://en.wikipedia.org/wiki/Two-phase_commit_protocol
[19:55] sustrik the whole point is that the 2nd phase is reduced to the bare minimum = 1 bit
[19:55] sustrik but it's still not perfect
[19:56] sustrik pieterh_: issue 100
[19:56] sustrik you've only seen it once?
[19:56] pieterh there is also http://en.wikipedia.org/wiki/Three-phase_commit_protocol, which is non-blocking
[19:56] pieterh sustrik: yes, just once
[19:56] pieterh could not reproduce it again
[19:56] sustrik hm
[19:56] pieterh it was perhaps related to the state of the TCP socket
[19:57] pieterh since it happened right after I stopped the peer node
[19:57] pieterh run handler, run gateway, stop gateway, stop handler, run handler *crash*
[19:57] pieterh handler is doing a bind
[19:58] mathijs I also found the "paxos commit algorithm". hmm... lots of reading :)
[19:59] sustrik pieterh_: so it happens when the application *isn't* terminating?
[19:59] pieterh sustrik: that's right, it happens at startup, I assume at bind time
[19:59] pieterh or recv
[20:00] sustrik ok
[20:08] sustrik pieterh_: strange, it looks like this kind of thing cannot happen unless there's a memory overwrite
[20:08] sustrik are you sure you are not overwriting memory in the test program?
[20:10] sustrik alternatively, it can be caused by using a bogus socket handle
[20:10] sustrik or a socket that was already zmq_close'd
[20:10] sustrik or using the same socket from multiple threads
[21:41] delaney kev009: i'm looking at writing a game server with zmq also
[21:54] delaney say you send 100 messages to a XREQ from a XREP and somewhere along the line (say message 5) the XREQ goes away forever. is there anyway to clear messages set for delievery to a specific envelope?
[21:54] delaney otherwise you have 95 messages on the queue forever
[21:57] delaney i see  individual high water mark in the api reference but don't see where to set it
[23:27] delaney sustrik are you there?