IRC Log

Friday October 28, 2011

[Time] NameMessage
[05:47] sontek Is there a system built on top of zeromq for doing tasks like celery + rabbitmq?
[05:47] sontek I'm using zeromq for pub/sub with websockets to do some realtime communication and so I'd rather use it than dropping rabbit in the mix
[05:53] guido_g is the salt project what you want?
[05:53] guido_g looking up url... stand by...
[05:54] guido_g https://github.com/thatch45/salt/wiki/What-is-salt%3F
[05:57] sontek It seems like it might be
[05:58] sontek lots of text there
[06:01] sontek basically just want to execute functions of python code in a queue
[06:01] sontek so for long running processes/blocking stuff I would execute it through this and return responses to my clients right away
[06:01] sontek rather than having my workers being tied up
[06:31] sontek guido_g: so definitely not what I needed but definitely something I'm going to use
[06:31] sontek guido_g: its like fabric but better
[09:02] mikko good morning
[09:03] sustrik 'morning
[15:19] babsher I am trying to write a peer to peer application, i was wondering what a good way to start would be
[15:19] babsher I was thinking that I could use pub sub sockets one for each of the peers
[15:20] mikko have you read the guide?
[15:20] mikko its a good start
[15:20] babsher I did read the guide but none of the patters really matched what i wanted
[15:21] babsher i have a collection of servers and they want to send messages to each other. I will know ahead of time where the message is going. each server should be able to get messages from any other server
[15:22] babsher Right now i dont care about reliability
[15:28] babsher would a push/pull work?
[15:31] cremes babsher: you need to figure out each operation that your application will perform
[15:32] cremes it's often the case that you need to use a combination of socket types (and patterns) to achieve
[15:32] cremes all of your goals
[15:32] cremes clearly push/pull work, but it's unclear if they will work for your application design
[15:32] mikko babsher: most likely multicast would work
[15:32] mikko assuming you have control over the network
[16:11] cremes sustrik: around?
[16:47] cremes quick C question for anyone lurking...
[16:48] cremes is it safe to cast a unsigned char to an int when returning it from a function call?
[16:48] cremes can i assume the upper 3 bytes will be cleared?
[17:00] Steve-o cremes: usually you get automatic int promotion
[17:01] cremes Steve-o: so do i need to do "return (int) function_that_returns_char();"
[17:01] cremes or is defining that function as returning int sufficient?
[17:01] Steve-o the latter
[17:01] Steve-o you can cast if you want though, obviously
[17:02] cremes great, thanks
[17:02] cremes btw, any feedback on my proposal to add zmq_msg_flags() to the api?
[17:04] Steve-o I don't use multi-part, so I just archived the mail, let me check
[17:05] Steve-o so you want to return the multi-part as an entire object
[17:06] cremes no, i want to be able to interrogate the message object to see if it is a label or part of a multipart message
[17:06] cremes without using zmq_getsockopt (which essentially does the same thing but via an unexposed api)
[17:07] Steve-o ok, makes sense
[17:43] sustrik cremes: hi
[17:43] cremes sustrik: hi... i just wrote a patch and am trying to get the test i wrote to work
[17:43] cremes let me gist it for you...
[17:43] sustrik ok
[17:44] cremes https://gist.github.com/1322866
[17:44] cremes the assert at line 43 blows me up
[17:44] cremes i thought labels were 4 bytes
[17:45] sustrik they are iirc
[17:45] sustrik what value are you getting?
[17:45] cremes line 42 is printing 17
[17:46] sustrik that's the GUID identity :(
[17:46] sustrik this is the problem with 3.0: it's a mix of backward-compatible semantics and new semantics
[17:47] sustrik 17 byte identity is a 2.1 feature, 4 byte identity in a new feature
[17:47] sustrik these two are combined in some way to make it somehow backwards comapatible
[17:47] sustrik it's a mess
[17:47] cremes yuck
[17:47] sustrik :(
[17:48] sustrik no idea how to fix it though, except for reverting to 2.1-like semantics or moving completely towards new semantics
[17:50] sustrik a big part of the problem is re-writing the examples in the guide if the semantics change
[17:50] sustrik it's a major undertaking
[17:50] cremes i agree
[17:51] cremes from a code simplicity and future-proofing standpoint, which do you prefer?
[17:51] cremes 2.1 semantics or 3.x semantics?
[17:51] sustrik the new one
[17:51] sustrik 3.x made consistently
[17:51] sustrik but from the maintenance perspective 2.1 is much better
[17:52] sustrik (no change to guide, no porting of existing apps etc.)
[17:52] cremes i will work on updating docs... i prefer 3.x too :)
[17:52] sustrik i think the compromise could work as follows:
[17:52] sustrik 1. revert 3.0 to 2.1-compatible semantics
[17:53] sustrik 2. implement new system in a complete and consitent way
[17:53] sustrik 3. push it out as a new major version in a year or two
[17:54] minrk the only disadvantage I see there is that while it makes the 2.1-3.0 transition easier, it actually makes the 3.0-4.0 transition harder
[17:54] sustrik true
[17:54] sustrik any better idea?
[17:54] indygreg at the point you revert 3.0 to 2.1-compatible semantics, what's the major difference in 3.0 at that point? could it be 2.2 or 2.3?
[17:55] sustrik there's also the C API change
[17:55] indygreg I guess the change in zmq_msg_send() is important
[17:55] sustrik that's mostly invisible to languages other than C
[17:55] sustrik but still, it breaks backward compatiblity
[17:55] sustrik of C apps
[17:56] cremes sustrik: which semantics do you want to reverse? eliminate LABEL and use empty message as delimiter again?
[17:56] sustrik yes
[17:56] sustrik and 17-byte IDs
[17:56] sustrik exacrly as before
[17:56] cremes i don't like moving backwards
[17:56] cremes i think LABEL is a good new addition
[17:56] sustrik in theory
[17:56] cremes i'd hate to lose it
[17:56] indygreg you have to cross the bridge sometime
[17:57] sustrik ah, you can use the 4.0 repo
[17:57] sustrik that's not the problem
[17:57] cremes if you want to take a step back, then i would want the ability for user programs to *set* flags on messages
[17:57] minrk I think treating ROUTER/DEALER as obviously legacy socket types might help, politically anyway
[17:57] sustrik the problem is how to migrate the existing ecosystem
[17:57] indygreg the repo clearly marked as unstable instead of beta?
[17:57] cremes because i want to be able to easily differentiate between routing envelope messages
[17:57] cremes and the message body in a multi-part setup
[17:58] sustrik minrk: yes, that's done
[17:58] sustrik cremes: try using XREP/XREQ in your test
[17:59] sustrik they should use the new system
[17:59] minrk ROUTER/DEALER allow people who haven't figured out how to build their app without IDENTITY (e.g. me) a path forward, so they can start using 3.0, and move to 3.0-style semantics when they figure it out
[17:59] sustrik as opposed to ROUTER/DEALER which use the old one
[17:59] cremes brbg
[17:59] sustrik :|
[17:59] minrk for instance, my app will be much more complicated using 4.0 than it is in 2.1
[17:59] sustrik minrk: ack
[18:00] minrk because it removes a significant amount of logic from zeromq, and puts it at the application level
[18:00] sustrik but afaiu what you are using identities for is just to route the messages to the right peer
[18:00] sustrik you don't rely on connection with identity storing messages even though the peer is offline/dead
[18:00] sustrik right?
[18:00] minrk correct
[18:01] indygreg sustrik: that guarantee was dropped in 3.0, right?
[18:01] indygreg unroutable peers are silently discarded?
[18:01] sustrik indygereg: it's same in 3.0 as in 2.1
[18:01] indygreg I thought 2.1 leaked memory on durable sockets and 3.0 didn't [when the peer goes away]
[18:01] sustrik minrk: that shouldn't be a problem, we can design a socket type for that
[18:02] sustrik what really has to go is the " storing messages even though the peer is offline/dead" stuff
[18:02] minrk ah, sure
[18:03] minrk that doesn't affect me - it's just the loss of settable routing identity is painful
[18:03] sustrik ack, i am aware of that
[18:03] sustrik so, i think my plan will help you
[18:03] sustrik reverting back to 2.1 style behaviour would not hurt you
[18:03] indygreg well, it just means you have to bake it into your application's message protocol, not as part of a multi-part message prefix. that seems semantic
[18:04] sustrik and we'll have 1-2 years to come up with a nice design for a routing socket
[18:04] jond sustrik: would it be possible to release a backward compatible 3.0 and a 4.0 simultaneously? How close is 4.0? People could choose the level of pain?
[18:04] minrk right, it will make 3.0 actually better for me
[18:04] sustrik minrk: please say so on the ML
[18:04] minrk but it means I can't get my app half-way to 4.0
[18:04] sustrik minrk: doesn't really matter
[18:05] sustrik 4.0 is far away
[18:05] sustrik jond: it'll take time
[18:05] minrk so it's a short term solution, that will make the 4.0 transition significantly more challenging.
[18:06] sustrik not necessarily, we'll have a lot of time to design the routing socket in such a way as to make the transition simple
[18:06] sustrik the problem at the moment is that 3.0 was released somewhere in the middle of the transition
[18:06] sustrik which makes it a bit creaky
[18:06] minrk ok
[18:07] sustrik historic note: 3.0 was released because subscirption forwarding was ready
[18:07] sustrik and we wanted to make it available
[18:07] sustrik however, we haven't considered the implications of the code already merged to req/rep/router/dealer subsystem
[18:07] minrk if there's a good answer for identity-based routing apps in 4.0, then I think a lot of the pain will be addressed
[18:08] minrk I know there are various users who rely on it
[18:08] sustrik jond: it's pretty far away; i would actually rather discard 4.0 as a project and move to topic branches instead
[18:09] sustrik minrk: there's a socket type in 4.0 with following properties:
[18:09] sustrik 1. it reports connections and disconnections
[18:09] indygreg sustrik: +1 to moving to topic branches
[18:09] minrk and the 4.0 ROUTER aka GENERIC is not ideal, since it requires significant application-level handshaking to get similar behavior
[18:09] sustrik ah, you see it
[18:09] sustrik well, what we can do is to wrap GENERIC into another socket type
[18:09] minrk I have already implemented one part of the application with it, to see what it looks like
[18:09] sustrik that would do the handshaking under the hood
[18:10] indygreg and get rid of the redundant Git repository clones while you are at it :)
[18:10] minrk that would be tremendous
[18:10] minrk I must admit, that there is pressure in Python to have logic in zeromq (or sufficiently general to be in pyzmq), because there is great cost to Python code over C code
[18:11] sustrik minrk: yes, agreed
[18:11] sustrik the socket types are an open set
[18:11] sustrik so if you define special messaging patterns these can be gradually incorporated
[18:11] minrk not just in execution time, but in the whole GIL-grabbing business.
[18:11] sustrik yes
[18:12] sustrik the GENERIC socket is actually meant as a way to allow for building arbitrary messaging patterns on top of 0MQ
[18:12] sustrik once the pattern proves itself, it can be moved inside 0mq
[18:12] minrk ok
[18:13] sustrik indygreg: merging the repos would be nice, but there are some governance issues there
[18:14] sustrik for example, Pieter (who manages 2.1 and 3.0) is less demanding wrt paperwork
[18:14] sustrik ie. there are patches that are not signed-off there etc.
[18:14] sustrik so, if we are to move to a single repo
[18:14] indygreg sustrik: ahh. I wasn't aware of that. I can relate
[18:15] sustrik community should decide on the process, legal assurances etc.
[18:15] sustrik the maintainers would then be bound by the community decisions
[18:17] cremes need feedback on my C/C++... is this acceptable? https://gist.github.com/1322961
[18:18] cremes or is there a more clever C-way to do that?
[18:18] sustrik you can do it in a single step
[18:18] indygreg cremes: you don't need the if
[18:19] indygreg you could also do it in one statement, at the cost of readability
[18:19] cremes show me... i'm too rusty with the language
[18:19] sustrik return u.base.flags & (msg_t::label | msg_t::more);
[18:20] indygreg temp_flags = (u.base.flags & msg_t::label) | (u.base.flags & msg_t::more)
[18:20] cremes sustrik: i had that originally but msg_t::more is 128 while RCV_MORE (exposed to end users) is 13
[18:21] cremes we need users to be able to do zmq_msg_flags() & ZMQ_RCVMORE
[18:21] cremes they don't know about the cpp enums
[18:21] indygreg oh, right
[18:21] sustrik 13?
[18:21] cremes #define ZMQ_RCVMORE 13
[18:21] minrk sockopt enum
[18:21] sustrik ah, that's the option ID
[18:21] indygreg well, in a new major version of the API you can change the #define's :)
[18:21] sustrik it's not the bit in the msg_t
[18:21] cremes true....
[18:22] cremes should i redefine the values for RCVMORE and RCVLABEL to match the enums in msg.hpp?
[18:22] sustrik sound hacky
[18:22] cremes i don't care either way... we need to keep the zmq_getsockopt(sock, ZMQ_RCVMORE, ...) behavior the same
[18:22] cremes anyway
[18:22] sustrik there are socket options and message flags
[18:22] sustrik they are distinct things
[18:23] sustrik maybe ZMQ_MSG_MORE
[18:23] sustrik and ZMQ_MSG_LABEL
[18:23] sustrik ?
[18:23] cremes ok
[18:23] cremes and have them match the values of the enums?
[18:23] sustrik it's up to you
[18:24] cremes e.g. #define ZMQ_MSG_MORE 128
[18:24] cremes #define ZMQ_MSG_LABEL 1
[18:24] sustrik presumably
[18:24] cremes i *do* have 32 bits to play with... why not
[18:25] sustrik minrk, cremes, indygreg, jond: guys, i am thinking of proposing a change to the management of the 0mq project
[18:26] sustrik would be nice to get some feedback from you before i post it
[18:26] minrk ok
[18:26] sustrik the idea goes like this:
[18:26] cremes shoot
[18:26] sustrik 1. revert 3.0 to 2.1 semantics
[18:26] sustrik 2. make 3.0 the "next big thing"
[18:26] sustrik 3. dismantel 4.0 and move individual new features onto topic branches
[18:27] sustrik 4. possibly we can talk about merging the repo's but that's going to be hard from administrative point of view
[18:27] sustrik thoughts?
[18:28] mikko i would love to have 2.1 compatibility on wire with 3.x
[18:28] mikko but that is very unlikely to happen
[18:28] mikko also, people are already building stuff on 3.0
[18:28] sustrik well, not for pub/sub
[18:28] mikko changing the semantics completely now might be detrimental
[18:28] sustrik there are subscriptions passed upstream now
[18:28] sustrik for req/rep and friends it's doable imo
[18:29] cremes it's going to *potentially* burn some of the 3.x early adopters
[18:29] sustrik yes
[18:29] sustrik that's why i'm asking
[18:30] mikko how did we end up into this confusing situation?
[18:30] cremes mikko: because 0mq doesn't follow "semantic versioning"
[18:30] sustrik sustrik> historic note: 3.0 was released because subscirption forwarding was ready
[18:30] sustrik <sustrik> and we wanted to make it available
[18:30] cremes er, it *does*
[18:30] indygreg regarding burning, that can be prevented in the future by not attaching version numbers to things until they are truly ready
[18:30] sustrik <sustrik> however, we haven't considered the implications of the code already merged to req/rep/router/dealer subsystem
[18:31] cremes an api change requires a new *major* number
[18:31] sustrik so, that's why i won't to get rid of 4.0 and move to topic branches
[18:31] mikko cremes: that explains the version number
[18:31] minrk My main concern with 1. (as expressed above) is that the partial-upgrade does have some transition benefits, but if the code is too ugly to support 2.1-style ROUTER/DEALER, and 3.0-style elsewhere, that's not worth it
[18:32] sustrik minrk: the code would look exactly as in 2.1
[18:32] sustrik no change
[18:33] minrk right - I'm saying the partial upgrade in 3.0 makes the move to slightly eases the move to *4.0*
[18:33] jond could 3.0 just be abandoned and the necessary patches just ported to a 2.2? it'd be less confusing.
[18:33] mikko this is a confusing situation in any case
[18:33] cremes jond: not if we follow the semantic versioning rules http://semver.org/
[18:34] cremes any backward-incompatible change to the public api requires a major version increment
[18:34] sustrik jond: there's also the API change
[18:34] sustrik zmq_send(void*, size_t)
[18:34] sustrik and similar
[18:34] sustrik that's actually pretty nice and not worth of reverting
[18:35] sustrik the only thing i was refering to wrt revert was all this LABEL stuff
[18:35] sustrik minrk: that assumes you are sure how 4.0 is going to look like
[18:36] minrk quite true - I don't feel too strongly
[18:36] sustrik in any case, topic branches would prevent this situation from happening again
[18:36] sustrik each feature would live on a topic branch
[18:36] indygreg would it help to have an etherpad or wiki page to track all this?
[18:37] sustrik and will be merged when it's ready, not sooner
[18:37] sustrik indygreg: i guess disussion on the mailing list would be best
[18:37] sustrik i'll send an email
[18:38] jond sustrik: i think the fact the you as the main developer of the code are expressing concerns about 3.0 is a real red light on this stuff.
[18:38] sustrik well, it's more of a concern about the process
[18:38] sustrik it was the process that allowed this thing to happen
[18:38] sustrik so we should fix it
[18:38] mikko i think you are right
[18:38] mikko the multiple repos and different approaches doesnt work
[18:39] mikko imho
[18:39] sustrik the process basically haven't changed from the old days when nobody used 0mq
[18:39] jond mikko: we've been saying that for years. pieterh just said 'one git per ego'
[18:39] cremes i've never liked that multiple repos approach
[18:39] sustrik it's doesn't fit anymore
[18:40] mikko also, i disagree with patches to mailing list approach as well
[18:40] cremes it never did when it was so easy to *fork* the repo in the first place
[18:40] sustrik jond: that's the administrative issue
[18:40] sustrik there are couple of points we differ on with pieter
[18:40] sustrik e.g. requiring sign-off
[18:41] sustrik or: appying patches that hide the faulty behaviour
[18:41] sustrik etc.
[18:41] jond yes but when sub forwarding was done on master, 3.0 was pretty much forked and released, and it looks like that shouldnt have happened.
[18:41] sustrik probably not
[18:42] sustrik pieter wanted a new major release back then
[18:42] sustrik and he was right, there haven't been one for a year
[18:43] sustrik however, the way the development was done haven't allowed to release subscription forwarding and new API
[18:43] sustrik without dragging in the experimental stuff like LABELs
[18:43] jond well that's why we need these topic branches
[18:43] sustrik yes
[18:44] mikko not sure how this will affect people who are already building on top of 3.0
[18:44] sustrik that's a question
[18:44] sustrik but i guess the revert won't be that bad
[18:44] mikko the whole versioning makes things worse
[18:44] mikko even though 3.x is beta there is 3.0.2 out
[18:45] mikko which indicates that there has been multiple releases
[18:45] mikko and people would assume it's safe to build on
[18:45] sustrik there have been: beta releases
[18:45] jond mikko: it's still beta in the name
[18:45] sustrik it should be called rc or something
[18:45] sustrik no idea
[18:45] mikko rc is more stable than beta
[18:45] sustrik true
[18:46] mikko jond: imho the versioning is a clusterf*ck
[18:46] mikko 3.0.2 indicates several releases of 3.x
[18:46] mikko which no other project does
[18:46] sustrik how do they version beta releases then?
[18:46] indygreg 3.0.0-beta1
[18:46] indygreg or 3-beta1 for that matter
[18:46] mikko well, depends but 3.0.0-beta1, 3.0.0-beta2 is pretty common
[18:47] mikko which clearly shows the maturity is still 0.0 and it's Nth iteration
[18:47] sustrik i see
[18:47] mikko vs 3.0.2 which is second patch release of 3.0
[18:48] mikko http://www.zeromq.org/intro:get-the-software
[18:48] mikko branding:
[18:48] mikko Upcoming release 3.0
[18:48] indygreg you can also do 3.0.1 [beta] and just say it is a beta release. for RC's, if there is no code change, you can just remove the RC flag from the release notes
[18:48] indygreg some people vehemently oppose releasing the same bits as different versions, however
[18:49] mikko isn't that the point of release candidate?
[18:49] mikko there should be no changes before RC and stable
[18:49] mikko changes would require a new rc imho
[18:49] indygreg in theory. although some applications (things with a GUI) sometimes recompile to remove the RC from the displayed version number
[18:49] jond sustrik: are you going to ask on the list about removing labels then?
[18:50] mikko indygreg: rebuilding is fine
[18:50] mikko indygreg: i mean code changes
[18:50] sustrik jond: yes
[18:50] sustrik all 4 points
[18:50] sustrik additionally:
[18:51] indygreg mikko: we are in agreement. the point of an RC is to not change any significant code before release
[18:51] sustrik if we want to make changes to the administrative structure of the project
[18:51] indygreg significant==meaningful==pretty-much-everything-except-displayed-version-numbers
[18:51] sustrik for example, to merge the repos
[18:52] sustrik the community should define the rules and impose them on maintainers
[18:52] sustrik right now it's me having my own rules on libzmq
[18:52] sustrik and pieter having his own rules for zeromq2-1 and zerom3-0
[18:52] mikko sustrik: i don't see this ending in an agreement :D
[18:53] mikko the biggest problem i find that it's so hard to follow which commit is which
[18:53] sustrik yep, it's an ego problem
[18:53] mikko sustrik: there is no ego problem with one repo
[18:53] mikko there is no reason to have multiple repositories
[18:53] mikko which a version control system that supports proper branching
[18:53] sustrik legal concerns?
[18:54] mikko regarding?
[18:54] sustrik signing off for example
[18:54] mikko then get a CLA from people
[18:54] sustrik you mean transferring copyrights?
[18:54] sustrik yuck
[18:54] mikko do we list somewhere what a signed off patch means?
[18:55] sustrik yes, it's a tracking system for the code
[18:55] sustrik so, for example, i get a patch from you
[18:55] sustrik i modify it
[18:55] indygreg if there are legal concerns, there can be one "official"/distribution repo/project and another one for community involvement. commits would occur on the community repo and would be merged to the official dist as appropriate
[18:55] sustrik there are two sign-offs there
[18:56] sustrik if there's a legal problem we can track the provenance of the code that way
[18:56] sustrik it's not a big concern right now
[18:56] sustrik but may be in the future
[18:56] cremes sustrik: when you post your 4 points to the list, indicate what features would/could be rolled back
[18:57] sustrik cremes: ok
[18:57] mikko sustrik: this signed off patches also make things very cumbersome
[18:57] cremes that way we can get feedback from 3.x users if their favorite feature is nuked
[18:57] mikko creating patch for three repos, then sending it, resetting heads of three repos, wait for merges and pull back the changes again
[18:57] cremes btw, just sent my signed-off patch to add zmq_msg_flags to the list :)
[19:14] sustrik cremes :)
[19:14] sustrik mikko: iirc the signing off thing was introduced to linux after sco case
[19:14] sustrik 0mq can drop that policy
[19:15] sustrik but again it has to be a community decision
[19:22] tarcieri cremes: so I think there's a socket type that fits Celluloid a lot better than an exclusive pair
[19:23] cremes tarcieri: yeah? so what are you thinking?
[19:23] tarcieri cremes: pull
[19:24] tarcieri a mailbox is a unique destination that N senders send messages to
[19:24] tarcieri pull seems like a good fit for that
[19:24] cremes ok, so the senders would have push sockets
[19:24] tarcieri yes
[19:25] tarcieri that's actually a really good fit for how Celluloid works conceptually
[19:26] tarcieri mailboxes aren't bidirectional
[19:26] tarcieri they're just message sinks
[19:32] cremes perfect
[19:35] tarcieri I'm still trying to decide if I'm crazy for wanting to use Zookeeper though :/
[19:35] tarcieri might be some YAGNI
[19:35] tarcieri I want something like this eventually: https://github.com/uwiger/gproc
[19:46] cremes tarcieri: zookeeper for service discovery?
[19:48] cremes tarcieri: if you want to "keep it in the family" look at http://zguide.zeromq.org/page:all#Service-Discovery
[20:00] grantr ooo
[20:02] grantr quote: "In practice a broker should remove services that have no workers after some configurable timeout."
[20:02] grantr trivializing a non-trivial problem! ;)
[20:32] tarcieri cremes: I'd like to make it modular, but that looks cool

Web site design and content is copyright © 2010-2011 Peter Alexander, ProDataLab. Site content licensed under the Creative Commons Attribution-Share Alike 3.0 License. ØMQ is copyright © Copyright © 2007-2011 iMatix Corporation and Contributors. Contact us for professional support. ØMQ is free software licensed under the LGPL. ØMQ, ZeroMQ, and 0MQ are trademarks of iMatix Corporation. Terms of Use ? Privacy Policy