Wednesday June 8, 2011

[Time] NameMessage
[00:14] wayneeseguin Is that bash binding actually implemented somewhere or is it a joke pieterh ?
[00:15] wayneeseguin If I want to add support to a language that is not there yet, where do I look to do that?
[00:15] wayneeseguin Do I have to look at others for examples or is there a guide
[05:40] whack wayneeseguin: I think there's a protocol spec for zeromq - might be it
[06:55] jkp question: is there a recommended way to implement a many-to-many "broadcast" topology without using multicasting?
[06:55] jkp im thinking of trying to use zeromq to replace multicasting as a mechanism for broadcasting updates from our servers to our clients
[06:56] jkp in this scenario, all clients must listen to all servers, but id rather avoid a broker if possible (for variou reasons)
[06:57] guido_g ømq has devices for that kind of thing
[06:57] guido_g devices are small programms with exactly one job, forward messages according to one pattern
[06:57] jkp guido_g: it does? i read the docs but i didnt see any devices matching that pattern
[06:57] jkp are they new?
[06:57] guido_g no
[06:58] jkp can you name one in particular i should look up?
[06:58] guido_g there're two kinds of devices: the ones in the library itself and thn some executables
[06:58] guido_g <- lib version
[06:59] guido_g the executables do come with the v2 of ømq and are built by default, afair
[06:59] jkp ahh, it looks like ZMQ_STREAMER would work...but
[06:59] jkp that still requires a broker to run the device
[07:00] guido_g of course
[07:00] ianbarber jkp: if you want many to many without multicast you have to have a device, or you have to have every node know about every other node
[07:00] guido_g how else should it work?
[07:01] guido_g complete mesh would kill performance, on app level and on os level
[07:02] jkp ianbarber: k, i was getting that impression
[07:02] jkp basically im trying to cope with the following scenario...
[07:02] ianbarber jkp: that's less a limitation of zeromq and more one of the fact the communication channels are point to point
[07:02] jkp we have a number of parallel stateless servers, all doing the same thing and sharing state through the db
[07:03] jkp clients make requests and update data and then all other clients should know when certain updates have happened
[07:03] jkp we currently use multicasts and have the clients listen to the group / port and reassebmle the messages at the other end
[07:04] jkp problem is that multicasting is pretty unreliable in a lot of cases and also the code is complex
[07:04] jkp zeromq looked like it could help here
[07:04] jkp brb
[07:04] guido_g sounds like a case for (e)pgm
[07:05] ianbarber yeah, i think you'll find pgm and zeromq a lot less complex than you may have expected
[07:05] guido_g one of the best things in ømq is the pgm support
[07:05] guido_g because you get most of the mutlicast w/o most of the trouble
[07:08] jkp yeah, i did some playing around with epgm yesterday
[07:08] jkp it worked as i would expect...but its multicast based so i guess it just buys me reduced code complexity rather than removing the potential issues of a multicast based solution altogether
[07:09] guido_g you guess?
[07:09] jkp heh :)
[07:10] jkp ok, "it buys me reduced code complexity" :)
[07:10] jkp better?
[07:10] jkp :p
[07:10] guido_g
[07:11] jkp guido_g: thanks for the pointer :)
[07:11] jkp i shall do some reading about pgm
[07:11] guido_g ack
[07:11] jkp the problem we've had with multicasts as a transport is just getting customers to set their networks up correctly
[07:11] jkp a lot of times they dont configure their routers right etc and then of course the finger is pointed at our application stack for being crap
[07:12] jkp (when updates dont arrive as expected)
[07:12] guido_g it's a cruel world
[07:12] jkp heh
[07:13] jkp yeah, but its meant that for a long while we've been thinking we'd like to use something else...something easier to configure correctly from a customer perspective
[07:14] guido_g think about an application that has to deal w/ hunderts or thousands of endpoints
[07:15] guido_g an all gos belly up if *one* port is wrong
[07:15] guido_g do you really think that is better for you customer (or you)?
[07:19] ianbarber you could offer 2 modes, have a broker device and have all connect to that, or have multicast for higher performance setups
[07:19] ianbarber that's really just changing zeromq DSNs, and writing the broker
[07:20] guido_g not even that, if you can use the zmq_forwarder
[07:21] guido_g bust sill, lots of tcp connections in one process, that will slow down delivery significantly
[07:41] private_meta jond: ah good, you see online
[07:42] private_meta <-- gdb output on redhat when SIGTSTP signal is sent while in "zmq::poll"
[07:52] mrmermaid Can anyone please help on building libzmq on VS C++ 2010 Express ?
[07:53] mrmermaid Please point me to some direction as I can only build the .dll file but not the .lib
[08:20] ianbarber mrmermaid: mikko might be able to help. is there anything similar on, that may give you a clue
[08:21] mrmermaid Thank you, I have figured it out
[08:22] mrmermaid I gave up and install VSC 2008
[08:22] mrmermaid I checked, and in Properties/Linker/Advanced/Import Library, this is the value that is missing: $(TargetDir)$(TargetName).lib
[08:23] mrmermaid not sure if this is enough, but compile with 2008 should give a green, posting here so that everyone knows.
[08:35] mikko mrmermaid: that happens because of the conversion
[08:35] mikko mrmermaid: for some reason the 2008 project file -> 2010 conversion removes the import library
[08:36] mikko i will test CMake stuff soon to see if that helps on windows builds
[08:52] mrmermaid Thank you for your effort.
[09:04] mile hi guys,
[09:05] mile I'm getting a "Bad file descriptor
[09:05] mile nbytes != -1 (mailbox.cpp:242)
[09:05] mile " error
[09:05] mile even when I disabled binding
[09:05] mile any ideas what might cause it?
[09:05] mikko pieterh: are you here?
[10:14] pieterh re
[10:14] pieterh hi mikko
[11:21] mato hi pieterh
[11:21] mato pieterh: how was silicon valley?
[12:41] pieterh hi mato
[12:41] pieterh I was mostly in downtown SFO
[12:41] pieterh it's pretty intense, every cafe is filled with guys working on their starups
[12:41] pieterh &startups
[13:28] Seta00 hmm, I think I'm going to write a ZMQ Groovy wrapper
[13:29] Seta00 so you can have two Actors in two different machines communicating with the same Actor syntax
[13:29] Seta00 this is going to be interesting
[13:35] tproa i read that as "gravy wrapper"
[14:02] ianbarber literally every time i think 'oh i need a shortcode for that link' pieter has already made one :)
[14:30] Seta00 anyone here uses jzmq?
[14:34] pieterh ianbarber: lol, I'm sure you can think of some new ones
[14:34] pieterh Seta00: I know some folk who use jzmq, yes
[14:35] mikko pieterh: we should be all set with jira data now
[14:35] mikko just needs fiddling the projects around
[14:35] mikko (LIBZMQX) is the fixed one)
[14:35] pieterh mikko: did Atlassian fix the messed up comments?
[14:35] mikko yes
[14:35] pieterh oh, wow, great!
[14:36] mikko their importer plugin is broken
[14:37] pieterh so what's the next step, Mikko?
[14:38] mikko pieterh: renaming or deleting LIBZMQ
[14:38] mikko and renaming LIBZMQX to LIBZMQ
[14:38] pieterh let's go for it
[14:38] mikko i can do it this evening
[14:38] mikko or you can do it any time
[14:38] pieterh I can do it now
[14:38] pieterh mato: you around?
[14:40] pieterh I see people have been adding issues to libzmqx
[14:42] mikko pieterh: nope
[14:43] mikko pieterh: theyve been copied from LIBZMQ
[14:43] mikko i think
[14:43] mikko 216 - N
[14:43] pieterh ah, cause I didn't see them in libzmq
[14:43] pieterh is it possible to change the project key?
[14:44] mato pieterh: i was going to do it, just waiting for confirmation from mikko
[14:44] mato which just arrived
[14:44] mato pieterh: mikko: i'll do it now ...
[14:44] pieterh mato: ok, I've renamed the two projects and set an avatar / URL for the new libzmq
[14:44] pieterh you can take over...
[14:44] mato ok
[14:45] pieterh I'm going to create a czmq project and we can offer this to other binding authors who want it
[14:45] mato pieterh: yes, i've already put intro text into the dashboard telling people the tracker is open for any zmq projects
[14:46] pieterh cool
[14:49] mikko this concludes the migration?
[14:49] mikko apart from the https cert
[14:49] mikko https cert / custom domain
[14:49] mato mikko: almost, can you ask in the support request if they can re-key LIBZMQX to LIBZMQ?
[14:50] mato mikko: that and e-mail notifications which i will setup next ...
[14:59] mikko mato: i'll ask if they can rekey it
[15:00] mato mikko: this forum post suggests it can't be done easily:
[15:00] mato mikko: but maybe they know a way
[15:01] mikko mato: export and reimport ?
[15:02] mikko i don't see why that wouldnt work
[15:02] mato we'll see what they say
[15:02] mato i'll leave the "please don't add issues" banner up for now
[15:04] pieterh mato: afair from previous use of Jira, it can be done with an export and import and manual edit in between
[15:04] pieterh mato: IMO that banner should go, it'll confuse people
[15:04] mato pieterh: not if we're going to do one more export/reimport
[15:05] mato pieterh: the only problem is, this is all hosted and only atlassian can do import/export it seems
[15:05] pieterh mato: I guess so, but I'm sending people to that page now for czmq as well
[15:05] pieterh ah, right
[15:05] pieterh bleh
[15:05] mato pieterh: that's why this is all taking so long ...
[15:05] mato pieterh: otherwise we'd have had it all sorted ages ago.
[15:05] pieterh that's why there was no import/export option anywhere
[15:05] mikko it's a trade-off
[15:05] mikko takes longer but we dont need to host it
[15:06] pieterh well, notifications seem to work
[15:06] pieterh finally, this makes it worthwhile :)
[15:07] mato notifications?
[15:07] mato those aren't even set up yet :)
[15:09] mato pieterh: anyhow, so we need to wait for atlassian to do the re-key
[15:09] pieterh yeah
[15:09] pieterh mato: default notification scheme works more or less
[15:10] mato pieterh: you're getting emails? i thought that wasn't turned on yet
[15:10] pieterh I didn't check it in detail but it's sending me emails for my own issues, which is fine
[15:10] mato ok
[15:10] pieterh just select 'default notification scheme' for project
[15:10] mato right
[15:10] pieterh github really don't get it
[15:10] pieterh how can smart people be so stupid
[15:10] mato github for some reason intentionally want just a tracker for small (a few people) projects
[15:11] pieterh ah, yes, it's not open source, of course
[15:11] mato neither is jira :-)
[15:11] pieterh github support explicitly said they won't do email notifications cause they don't like email
[15:11] pieterh indeed, they're reducing the email notifications already in place, it seems
[15:11] mato pieterh: how was the trip to california?
[15:12] pieterh mato: it was great
[15:12] pieterh there are a lot of folk using 0MQ in California and Washington state
[15:12] mato real users, cool
[15:12] pieterh real users
[15:12] mato shame they're so far away...
[15:12] cremes pieterh: are they targeting any particular platforms?
[15:13] pieterh I kind of summarized it here:
[15:14] mato ?
[15:15] mato that doesn't tell me anything about our users :-)
[15:15] mikko this makes me wonder about 3.0
[15:15] mikko if we have a lot of active users
[15:16] pieterh well, the point is the use case
[15:16] pieterh it's cloud and mobile
[15:16] mato what use case? there's no use case in there
[15:16] mato stuff on the cloud talks to ... other stuff ... :-)
[15:16] pieterh you want specific verticals?
[15:16] mikko what is happening to identities in 3.0?
[15:17] pieterh mikko: nothing yet, afaik
[15:17] mato no, i just asked what are users are mostly doing, since you met with a bunch of them
[15:17] cremes i'm curious about linux versus osx versus windows versus os390, etc
[15:17] mato ruby vs. python vs. perl vs. java vs....
[15:17] cremes that too
[15:17] pieterh ok, specifics: mostly Linux, mostly Python, Ruby, and C/C++, and mostly EC2 or similar
[15:17] mato mostly web 2.0 startups or also others?
[15:18] pieterh Microsoft are looking at 0MQ for their clouds, so that's obviously not Linux but they're the only ones
[15:18] mikko we should have sockopt for being able to disable identities on pub side
[15:18] pieterh mostly startups that want to build massive cloud services
[15:18] mikko or maybe a callback when client connects with identity
[15:18] mato right, so we're fashionable :-)
[15:18] pieterh think of... mongohq, simplegeo, etc. (these don't use 0MQ but are typical use cases)
[15:18] pieterh distributed cloud databases
[15:18] pieterh that's what mobile apps need
[15:18] cremes great
[15:18] pieterh the key points are cost and scalability
[15:19] pieterh it has to be extremely cheap and extremely scalable
[15:19] pieterh clouds make their money from advertising, basically
[15:19] mato hmph
[15:19] pieterh meaning you have to handle a million users to make $1,000 profit (or somesuch)
[15:20] pieterh the figures are not real, but you get the point
[15:20] mato ok, any people running real businesses? :-)
[15:20] pieterh yes, but all in startup mode afaik
[15:20] pieterh the cycle is obviously 12-18 months and 0MQ/2.x has only been stable for 6 months
[15:20] cremes any area in particular where the established community should focus?
[15:21] cremes docs..
[15:21] pieterh cremes: yes
[15:21] cremes stability...
[15:21] cremes new features
[15:21] pieterh think clouds talking to mobile devices...
[15:21] pieterh security
[15:21] pieterh https
[15:21] pieterh that's the number one request
[15:21] cremes ok
[15:21] pieterh number two is probably "should not crash when given bad data"
[15:22] pieterh there's no number three afaict
[15:22] cremes yes, it needs to be impervious to fuzzing
[15:22] pieterh ye
[15:22] pieterh s
[15:22] mikko and prevent other DoS possibilities
[15:22] cremes anything about javascript or 0mq in the browser?
[15:22] pieterh cremes: nope
[15:22] mikko like being able to disable explicit identities on server-side
[15:22] cremes interesting...
[15:22] pieterh I'm sure some people want to get 0MQ natively in the browser
[15:23] pieterh there's an actionscript stack
[15:23] pieterh but it's moot... the real problems are security and firewalls
[15:23] cremes sure, but apparently none of the startups (think "lots 'o startup cash") want it so it may not get much attention
[15:23] pieterh oh, request number 2 was actually subports, now that I think about it
[15:24] pieterh it can be really annoying to have to open 5 ports across a firewall
[15:24] cremes agreed
[15:24] cremes but it hasn't really hurt my use of 0mq yet
[15:24] cremes so i don't feel that particular itch :)
[15:24] pieterh the consensus in SFO was "named subports in one process"
[15:25] pieterh anyhow, the interesting news is that there's nothing else like 0MQ for this space
[15:25] pieterh people like dotcloud started with RabbitMQ then switched to 0MQ
[15:25] pieterh some others considered writing their own TCP fabric, then abandoned that
[15:25] cremes yes, most people i have talked with have tried amqp and abandoned it
[15:26] cremes some went 0mq, others with homegrown
[15:26] pieterh I expect we'll get a lot of amqp refugees in about 6-12 months as amqp/1.0 emerges
[15:26] cremes very few have gone with commercial (29west, tibco, etc)
[15:26] pieterh the main questions will be stuff like "how do I do topic routing"
[15:26] pieterh we may want to provide standard answers to that
[15:26] cremes answer: with a device
[15:26] pieterh i don't see subscription upstreaming working soon
[15:27] pieterh well, a device that acts like an amqp topic exchange
[15:27] pieterh not trivial, but not rocket science either
[15:27] cremes right
[15:27] cremes i have some vague interest in using 0mq as a foundation for an actor library
[15:28] pieterh one thing I noticed is that people stick with the version of 0MQ they first use in production
[15:28] cremes pretty much everything i build is a small single function client (or server)
[15:28] cremes which maps to the actor model quite nicely
[15:28] pieterh do you mix them together using inproc?
[15:28] cremes i use whatever transport makes sense except for the pgm-based ones
[15:28] pieterh right
[15:28] cremes so yes, inproc is used
[15:29] pieterh its a shame we didn't collect useful devices (and device designs) for reuse
[15:30] cremes that may happen over time; it's not like the ideas aren't still out there
[15:36] nilsh is a libzmq crash expected on icmp "Destination Net Unreachable" ?
[15:37] cremes nilsh: no
[15:37] cremes did it print an assertion?
[15:37] nilsh errno == ECONNREFUSED || errno == ECONNRESET || errno == ETIMEDOUT || errno == EHOSTUNREACH (tcp_connecter.cpp:300)
[15:42] nilsh maybe the errno for "Destination Net Unreachable" should be added there to be ok?
[15:42] mikko yes
[15:48] nilsh ok i will create a bug report for ths
[17:39] mikko jira looks good now
[17:39] mikko mato: can you check?
[17:46] yukonbob hello, #zeromq
[17:47] mato mikko: checking...
[17:47] mato mikko: seems mostly fine
[18:04] mato mikko: ok, i've deleted the banner asking people not to add stuff
[18:04] mato mikko: i think we're good to go
[18:04] mikko cool
[18:04] mikko still need to handle the domain at some point
[18:04] mato yes, later