Thursday January 6, 2011

[Time] NameMessage
[06:52] Malinga hi all i have a question to be asked
[06:53] Malinga about how to use OpenAMQ or ZeroMQ to deliver market data (quote server)
[06:53] Malinga *effectively
[06:56] Malinga hi any one there to help my technical question
[07:00] sustrik what's the problem?
[07:02] Malinga We are developing a Quote Server which will be handel the market data and deliver it to the end client
[07:02] Malinga we are think of using a SOA architecture for this
[07:03] Malinga So can u tell me some information how to use OpenAMQ to make this happen
[07:04] guido_g hrhrhr
[07:04] Malinga i used openamq 3 years back in my previous company as a middleware
[07:04] guido_g sorry
[07:04] sustrik openamq or 0mq?
[07:04] Malinga but now i want to use it to deliver market data effectlvely and use this openAMQ as a quote server
[07:04] Malinga any idas ?
[07:04] guido_g you did realize that zeromq is not related to openamq?
[07:05] Malinga if zeroMQ is better approach i can use ithat even
[07:06] guido_g no its not better
[07:06] guido_g its different
[07:06] Malinga any architecture gudes that u can give sustrik ...
[07:08] Malinga so can i use zero MQ for my senario
[07:08] sustrik Malinga: on the quote publisher open a PUB socket
[07:08] sustrik send the quotes to the socket
[07:08] sustrik on clients, open a SUB socket
[07:08] sustrik do your subscriptions
[07:09] sustrik and recv() the messages
[07:09] sustrik that's all
[07:09] Malinga listning
[07:09] Malinga this is via openAMQ right ?
[07:09] sustrik no, it's 0mq
[07:09] Malinga sorry
[07:09] Malinga ok
[07:10] Malinga so u think we can use the 0MQ to our market data quote server ?
[07:10] sustrik yes
[07:10] Malinga cool
[07:10] sustrik have a look at 0mq guide
[07:10] Malinga do u have some high level architecture on this is possible
[07:10] sustrik there's an example of -- i think -- distributing weather forecase
[07:11] sustrik forecast
[07:11] Malinga hmmm
[07:11] sustrik it's basically the same as you need for stock quotes
[07:11] Malinga so like in openAMQ do we have to have per symbol queues sorry i dont have much ida about 0MQ
[07:12] sustrik what do you need it for?
[07:12] sustrik fair balancing between individual symbols?
[07:14] Malinga u mean per symbol queues ?
[07:16] Malinga what is the better open 0MQ or OpenAMQ for my quote server
[07:17] sustrik openamq is a legacy technology
[07:17] sustrik so i would recommend 0mq
[07:17] sustrik it's also much easier to use
[07:17] sustrik and you don't need the broker
[07:17] sustrik which makes administration easier
[07:17] Malinga cool
[07:18] Malinga but our entire application (end clients) is in Java and hope to write back end also in java
[07:18] sustrik sure
[07:18] sustrik check java 0mq bindings
[07:18] Malinga u have java wrappers right ?
[07:18] sustrik yes
[07:19] Malinga realiable as C++ ? i mean java wrappers ?
[07:19] sustrik it's wrapper on top of C
[07:19] sustrik so it's the same thing
[07:19] Malinga JNI /
[07:19] Malinga JNI ?
[07:19] sustrik yes
[07:19] Malinga hmmm
[07:19] Malinga ok
[07:19] Malinga let me read 0MQ
[07:19] Malinga and come back to u in a few days
[07:20] sustrik you are welcome
[07:20] Malinga anyway thanks so much for all those informatin, i still can remember u helped me allot (u may not rememebr now) few years back also
[07:21] Malinga when im doing another system with openAMQ
[07:21] Malinga it was a gr8 sucess
[07:21] Malinga thanks
[07:21] Malinga and bye for now
[07:21] sustrik i remember your nick :)
[07:21] sustrik see you
[07:42] CIA-21 zeromq2: 03Min Ragan-Kelley 07master * r08cd2ce 10/ (AUTHORS include/zmq.h include/zmq_utils.h):
[07:42] CIA-21 zeromq2: resolve "function declaration isn't a prototype"
[07:42] CIA-21 zeromq2: change two declarations in headers to form f(void); instead of f();
[07:42] CIA-21 zeromq2: which eliminates the warnings when compiling against zeromq
[07:42] CIA-21 zeromq2: Signed-off-by: MinRK <> -
[09:14] mikko good morning
[09:29] sustrik morning
[10:13] jugg After calling zmq_send(..., ZMQ_SNDMORE), is there a way to abort the transfer without sending the final message part? eg. If an application error occurs disallowing sending the final part(s), is closing the socket the only way to recover?
[10:16] jugg A reset socket option would be nice...
[10:18] mikko you mean reset as in send empty message part or a part flagged as "reset" ?
[10:32] jugg I mean, abort the sending of the message parts (don't send them over the wire). Which they aren't at this point. However, currently either you have to send another message without sndmore set, thus the entire bad message is put on the wire, or you have to close the socket.
[10:35] sustrik jugg: what are you trying to achieve?
[10:35] sustrik how come the message is "bad"?
[10:36] jugg the message is incomplete... the application code generating the message parts can not finish.
[10:37] jugg Of course, the application code could wait until the entire message is built, but that isn't always interesting.
[10:38] sustrik so what happens?
[10:38] sustrik something like timeout?
[10:38] jugg the application would like to go on to start building another message, but it can not send a new message until the old one completes. So it either has to allow a bad message to be sent, or close the socket.
[10:39] sustrik i mean, how come the message is bad?
[10:39] jugg it is incomplete
[10:39] sustrik why are you sending it then?
[10:40] jugg because it is sent as it is built.
[10:40] sustrik hm
[10:40] sustrik that's not what multi-part messages are for
[10:40] jugg obviously a solution is to wait to send until it is fully built, but...
[10:40] sustrik the goal is not to hold the list of message parts on behalf of the user
[10:41] sustrik it's more like gather/scatter array
[10:41] sustrik allowing for zero-copy of data that is not placed in contiguous memory
[10:41] sustrik i would suggest holding the message parts in your app
[10:41] jugg yes, I'm aware of what it is designed for... But that doesn't mean I won't try to extend its use case...
[10:42] sustrik and sending only when the message is complete
[10:42] sustrik sure, go on :)
[10:43] jugg a socket reset option would help. - for other reasons as well, but I know I'll get more flac for those.. :)
[10:43] jugg btw, I'll probably hit you up for erlzmq access tomorrow... it seems to be working well.
[10:44] jugg Still a few more test cases to check.
[10:44] sustrik well, my concert is that the option would bind the API to particular implementation
[10:44] sustrik ie.
[10:45] sustrik currently the code can be rewritten/optimised in such a way that message parts are sent over the wire
[10:45] sustrik even if the message is not fully sent
[10:45] sustrik and cached on the receiver side
[10:45] sustrik if the connection breaks, the cached message parts would be discarded
[10:46] jugg you'd have to send an abort message then, so the receiver could discard the incomplete parts.
[10:46] sustrik the rollback option would then require to extent the wire protocol
[10:46] sustrik exactly
[10:46] jugg I see
[10:46] sustrik the interactions are quite complex
[10:47] sustrik now, doing that just to not force user to keep a simple list of message parts seems an overkill
[10:48] sustrik in more generic terms, 0mq was designed to implement the minimal possible subset of features
[10:48] sustrik everytihng that can be build on top was deliberately eliminated
[10:48] sustrik so, if you want this kind of thing, just go on, implement a wrapper on top on 0mq
[10:49] sustrik or maybe a whole stack layer that would provide all kind of additional functionality
[10:49] jugg Well, that view point simplifies a problem I was having with the erlang bindings... :)
[10:53] jugg I have to run. Thanks for the explanation. Combined with the discussion we had some time ago where I wanted zmq to send each part over the wire as it was received, I might be starting to 'get' it :)
[10:54] sustrik it's simply avoinding the feature creep
[10:54] sustrik see you later
[11:38] parik Hi
[11:40] sustrik hi
[11:40] parik I was going through 0MQ src and found that it is based on PGM protocol.So i studied first PGM protocol and it was intersting need to know about multicasting
[11:41] parik Like how no acknowledgment for successful delievery and negative acks for data loss and duplication of packet by the router
[11:42] parik and hence reduce Server Load and Network Load
[11:43] parik Do i need to download open PGM to make 0MQ work
[11:44] parik I am not getting the defintion for pgm_init..I looked around in PGM also
[11:44] parik Hi Sustrik are you there
[11:44] sustrik parik: 0mq is multi-protocol
[11:45] sustrik so you have tcp as well as pgm
[11:45] sustrik if you don't need mutlicast you can just ignore pgm
[11:45] sustrik if you need it, openpgm is packaged with 0mq (see foreign/openpgm subdir)
[11:46] sustrik to build 0mq *with* pgm you need to add an option to confiuger:
[11:46] parik Thanks Sustrik.
[11:46] sustrik ./configure --with-pgm
[11:46] mikko does openpgm maintain ABI?
[11:46] parik ok.Thanks alot.
[11:46] parik What is ABI
[11:46] mikko i'm wondering if packaging openpgm with 0mq makes sense in long-term
[11:47] mikko application binary interface
[11:47] sustrik it would be better to have an separate openpgm package
[11:47] sustrik but so far it doesn't look like that
[11:47] parik Thanks Mikko
[11:47] mikko the main problem with current implementation being hard to upgrade and added maintenance for 0mq project
[11:48] sustrik the actual fact, afaiu, is that most people are using openpg via 0mq
[11:48] sustrik mikko: right
[11:48] sustrik it's not obvious how to solve it though
[11:48] mikko so there are situations where openpgm is supported on certain platform + compiler combination but cant be compiled with --with-pgm
[11:49] parik You mean to say it's better to use open pgm pakage then that comes with OMQ
[11:49] sustrik i am aware of that
[11:50] sustrik but on the other hand requiring users to install scons and build openpgm by hand
[11:50] mikko there are two ways to change it: a) decouple pgm completely from 0mq and have users install it separately b) build openpgm using their scons builds and just link against artifacts
[11:50] sustrik manage the dependencies
[11:50] sustrik etc.
[11:50] mikko the latter adds dependency to scons
[11:50] sustrik could be actually a barrier to adoption
[11:50] mikko as does the former
[11:50] parik So pgm that is bundled with 0MQ will be potable to all OS and architectures
[11:50] mikko sustrik: in the long run 99% of the users will get 0mq via their distro
[11:50] mikko well, maybe not that large amount but high percentage
[11:50] sustrik yes
[11:51] sustrik so the goal is to have openpgm packages for all distros and 0mq packages for all distros
[11:51] sustrik we are quite far away from that afaics
[11:51] sustrik the current system is the next best thing imo
[11:52] mikko im not sure if dependency to scons is a large issue
[11:52] mikko i see python as larger issue but that is already a dependency
[11:52] sustrik huh
[11:52] sustrik is it?
[11:52] mikko yes
[11:52] sustrik where?
[11:52] sustrik openpgm?
[11:52] mikko yes
[11:52] sustrik oh my
[11:53] mikko
[11:53] sustrik any idea of what is it used for?
[11:53] mikko i don't but i can easily check
[11:53] mikko foreign/openpgm/@pgm_basename@/openpgm/pgm/version.c: ../foreign/openpgm/@pgm_basename@/openpgm/pgm/
[11:53] mikko python ../foreign/openpgm/@pgm_basename@/openpgm/pgm/ > $@
[11:54] sustrik hm
[11:54] sustrik let me see
[11:54] mikko
[11:54] mikko github doesn't show lines so i can't link you directly
[11:54] mikko but should be easy to find
[11:56] sustrik the script generates the version name
[11:56] sustrik possibly we can replace it by a hardcoded one
[11:56] sustrik and thus drop the python dependency
[11:57] mikko but then again, there hasn't been complaints about python dependency
[11:58] sustrik the problem imo is that the build comes very early on in adoption process
[11:59] sustrik so, say if you find a bug in 0mq, you are already using it
[11:59] sustrik so it make sense to report the bug
[11:59] sustrik or even fix it
[11:59] sustrik however, if you are building it
[11:59] sustrik it means you are using 0mq for 5 mins at the moment
[12:00] sustrik if there's a problem, the simple solution is to just let it be
[12:00] sustrik and use something else
[12:00] mikko hmm
[12:00] mikko i agree and disagree
[12:01] mikko i agree that build issues will throw off people
[12:01] mikko i disagree that clear dependency will throw off people, especially when it's build-time only
[12:01] mikko we require autoconf 2.61 as well which is not available everywhere
[12:02] sustrik ack
[12:02] mikko brb, fetching lunch
[12:02] sustrik otoh, autoconf is a standard package
[12:02] sustrik openpgm is not yet a package
[12:02] sustrik you have to build it by hand
[12:02] sustrik etc.
[12:02] sustrik bon appettit
[12:13] parik Sustrik.Thanks alot.Yes I got pgm package inside foreign subdir.I am going to do code walkthrough before i take up and understands 0MQ
[12:13] parik Thanks alot Sustrik once again
[12:14] parik Thanks mikko
[12:14] sustrik :)
[12:15] parik Do I am going in right direction.I always wnat to know as much as possible so as this learning will help me somewhere
[12:18] sustrik parik: have you seen the 0mq guide?
[12:18] sustrik it's worth of reading for the starters
[12:18] parik Yes ..
[12:19] parik Thanks Sustrik.Actually I have not yet gone through it completely
[12:20] sustrik ch1 & ch2 are the most important
[12:20] sustrik to grasp the concept
[12:20] parik When I come across 0MQ is based on PGM My entire focus goes to understand PGM first as I was not aware fo it earlier
[12:21] parik I will definitly go through ch1 and ch2
[12:21] sustrik parik: be careful with pgm
[12:21] parik Now i understands PGM to certain extent to start with
[12:22] sustrik mutlicast is only useful in special circumstances
[12:22] parik You mean to say shall i not concentrate much on this
[12:22] parik ok...
[12:22] sustrik in most cases tcp makes more sense
[12:22] sustrik pgm =very high volume pub/sub on LANs
[12:22] sustrik it needs quite a lot of administration
[12:22] parik But does not three way shaking makes it more congested in TCP
[12:22] parik ok..
[12:23] sustrik it requires good hardware etc.
[12:23] parik You mean to say pgm compatible routers
[12:23] parik upgraded hardware for performance
[12:23] sustrik in most cases you don't event want multicast traffic pass the router
[12:23] parik ok
[12:23] sustrik rather i meant switches with IGMP snooping
[12:24] Steve-o pgm routers are v. expensive
[12:24] Steve-o switches are pretty easy, standard procurves work well
[12:24] parik ok..So that's an expensive one
[12:25] Steve-o well, v.expensive for gigabit capacity, cisco support it on almost their entire product range, expensive = US$10,000
[12:26] parik oh...That's really expensive...
[12:27] parik So then i will start with TCP.But no ways going through PGM gives me some insight of what is it and how does it work
[12:27] parik I will start with TCP
[12:27] Steve-o 100mb is around $5,000 the closest I saw last check, slower speeds all the way down to $250 for DSL routers, etc.
[12:28] Steve-o consider PGM an administrative option rather than anything particular technical for development
[12:28] Steve-o that's how zeromq has been developed, choose a ZMQ_PUB socket and run it to the ground with TCP until you really need scaling
[12:28] sustrik right, you can start with tcp and switch to pgm when needed
[12:29] Steve-o consider TCP sockets are pretty much accelerated everywhere and PGM isn't anywhere
[12:29] parik Thanks for guidance and making me to go in right direction
[12:29] parik Yes I do agree
[12:30] Steve-o but when you need it, it's awesome :D
[12:30] parik I am a novice but got some useful tips from Sustrik and Steve
[12:30] parik :-)
[12:33] parik Just one thing I always want to work and write some protcols, creating some utility but i always wonder how can I start with
[12:35] Steve-o implementing your own network protcol or are we talking above zeromq?
[12:40] parik I mean to say something I can contribute to and involved myself as one of the developer
[12:41] Steve-o well there is plenty of scope with zeromq
[12:41] Steve-o a lot can be made on top, e.g. last value cache, certified delivery, transactional delivery, virtual circuits, http tunneling, encryption, authentication, proxying, etc
[12:43] parik Great..Since I am novice and don't know How I can also be felt great of doing/contributing something..Just I want to know How can I contribute...Need some guidance on this front
[12:44] Steve-o well it's usually easiest if you have a target architecture to develop for,
[12:45] Steve-o for example a last value cache could be simply integrating memcached from TCP/UDP to zeromq sockets
[12:45] keffo What's wrong with abstract logic? =)
[12:47] parik Steve I am really sorry ..I didnot get it
[12:49] Steve-o if you have a scenario to deploy something its easier than randomly creating something
[12:49] Steve-o unless you are into computer science theory I guess
[12:49] parik oh..
[12:50] parik Yeah I am in Comuter Science
[12:50] Steve-o there is one developer integrating zeromq with websockets for their app, that's pretty neat
[12:51] parik ok..
[12:52] Steve-o you can also pickup ideas from here:
[12:53] parik Thanks alot Steve..I think this will really help.
[12:55] parik Thanks Steve You have given me a starting point to start with..I
[12:55] parik am really grateful to you
[12:57] parik Thanks alot once again to Steve to Sustrik for guiding me and answering my doubts and given your precious time.
[12:57] Steve-o good luck, post something on a blog and let us read about it
[12:58] sustrik feel free to discuss your ideas on the mailing list
[12:58] parik Thanks alot Steve.I will make it sure...
[12:58] sustrik there's much more people there to comment then on irc
[12:58] parik oh...ok
[12:59] sustrik Steve-o: btw, your Tibco migration docs are really useful
[12:59] sustrik especially the code examples
[12:59] parik Thanks once again to Steve and Sustrik...I donot want to leave the discussion but i have to as I have to go but i will cherish the discussion with you people.You people are alot supported
[12:59] sustrik it would be great if these were widely accessible
[13:00] sustrik even if they are not perfect, it at least allows rv users to start somewhere
[13:00] Steve-o ideally dumping in github or somewhere
[13:00] Steve-o as I mentioned I already have C++ & Java examples but didn't include them
[13:01] Steve-o but honestly I have no idea how you want rv users to migrate no-single-point-of-failure architectures
[13:01] sustrik i would say the text can be dumped on a wiki page, while code examples can reside on github
[13:02] sustrik well, we can address that later on
[13:02] sustrik the important thing is make the initial docs accessible
[13:02] Steve-o that will make all the order routing developers happy
[13:03] keffo routing developer, now that's a career! :)
[13:03] sustrik it's matter of time imo, you can't do everything straight away
[13:04] sustrik so let them try 0mq, experiment with it, complain about it
[13:04] sustrik and hopefully solution would emerge
[13:04] keffo sustrik: Do you know if anyone has tried to bridge zmq with wcf?
[13:04] sustrik i think i dimly remember something like that
[13:05] sustrik but i may be wrong
[13:05] Steve-o keffo: order routing is rather dull, especially FX
[13:05] keffo Steve-o: my point yeah :)
[13:06] Steve-o 1,000 trades a day max, woohoo
[13:06] keffo sustrik: It's easy enough to just call a zmq api inside a wcf service, but a somewhat more native method would be very useful
[13:08] sustrik hm, tried googling?
[13:08] keffo yeah
[13:09] sustrik doesn't look like anything is available
[13:10] sustrik small project for someone to spend weekend hacking :)
[13:10] keffo hmm, rabbitmq seems to support it, and also federation services, that's totally awesome if they support full saml & adfs
[13:10] keffo not that it helps in any way I guess, exept perusal of their code maybe
[13:10] Steve-o presumably wcf 4 looks most interesting?
[13:11] keffo how do you mean?
[13:11] keffo if I did decide to try I would target the latest version obviously?
[13:11] Steve-o the original cl wrapper was designed around .net 2 for greater compatibility
[13:12] keffo I already have my loadbalancer ui in wpf, but there's something oddly appealing about hosting the loadbalancer directly in IIS, with the ui as a silverlight app
[13:13] keffo Steve-o: this would be a lib, using the zmq api, and nothing more
[13:31] keffo "The RabbitMQ .NET client is an implementation of an AMQP client library for C# (and, implicitly, other .NET languages), and a binding exposing AMQP services via Microsoft's Windows Communication Foundation (WCF)."
[13:31] keffo So, doable
[13:35] mikko
[13:35] mikko doesn't seem to be overly complicated to create custom bindings
[13:36] mikko then again, i really dont know anything about windows
[15:14] mikko sustrik: there?
[15:14] sustrik mikko: yes
[15:14] mikko where is hosted at?
[15:16] sustrik here in bratislava
[15:17] sustrik it's a box at the local ISP's datacenter afaik
[15:17] mikko cool, i noticed that the 404 handler is not very user-friendly:
[15:17] mikko for example
[15:18] mikko i can do a small page that shows "Did you mean: x"
[15:18] mikko not sure if that is a high priority though
[15:19] sustrik you have to speak to mato
[15:19] sustrik he's managing the box
[15:19] mikko where is mato nowadays btw?
[15:19] sustrik new zealand
[15:19] mikko contract work?
[15:19] sustrik he's half slovak, half new zealander
[15:19] sustrik on his holiday atm
[15:20] sustrik i think he should be available shortly
[21:46] Namei Hello
[21:54] sustrik hi
[22:01] mikko hi
[22:01] Namei how are u ?
[22:02] mikko im fine thank you
[22:02] mikko yourself?
[22:04] Namei the same :)
[22:06] Namei i came in a english server to perfect my english, but it's hard lol
[22:20] Namei good night