IRC Log

Wednesday October 12, 2011

[Time] NameMessage
[00:00] mkoppanen rather than immediate
[00:00] mkoppanen or maybe outbound mail servers are going through maintenance or something similar
[00:00] nanodocumet but the message will be queue right?
[00:00] mkoppanen yes
[00:01] mkoppanen it all depends on how you handle it
[00:01] mkoppanen which language are you using?
[00:01] nanodocumet PHP.
[00:01] mkoppanen for the worker processes as wlel?
[00:01] nanodocumet I have installed zeromq and php bindings on ubuntu.
[00:01] mkoppanen as well*
[00:02] nanodocumet yes, I am planning on using PHP for all.
[00:02] nanodocumet I use Kohana framework.
[00:02] nanodocumet (that's where I heard about zeromq).
[00:03] nanodocumet I already have some scripts (running using crontab) to send the emails.
[00:04] nanodocumet I was planning on adding the messages (tasks for sending the emails with all the information) from Kohana.
[00:04] mkoppanen i guess samsoir is the champion of zeromq in the kohana community
[00:04] nanodocumet yes!
[00:04] nanodocumet he mentioned it!
[00:05] mkoppanen i used to work with him back in the day
[00:05] mkoppanen in london
[00:05] mkoppanen you could start testing with simple PUSH/PULL pattern
[00:05] nag nanodocumet: fyi, the PHP implementation has worked well for me. we pass data between a set of C daemons and PHP via zmq msgs.
[00:05] mkoppanen and extend from there to suit the specific scenario
[00:06] mkoppanen nag: good to hear
[00:06] nanodocumet nag: that's what I've been reading, and what I really like.
[00:06] nag that first moment when things 'just work' is refreshing
[00:07] mkoppanen so, now that we have some php users here i got a question for you guys
[00:07] mkoppanen would you find useful if there was a tool that allowed communication with zmq
[00:07] mkoppanen and would manage worker processes over fcgi
[00:07] mkoppanen so that php-fpm could be used to manage workers
[00:07] mkoppanen and the requests would look like just like any other fastcgi request
[00:09] nanodocumet I think that's what samsoir mentioned.
[00:09] nag mkoppanen: I believe so. Does the PHP implementation not work with something like nginx then?
[00:09] mkoppanen nag: it does
[00:09] mkoppanen nag: but what i see often that people write worker processes in php
[00:09] mkoppanen which is not ideal but probably works
[00:10] nag ah, I see... I wouldn't personally find it useful in my current work
[00:10] mkoppanen if you could use php-fpm to manage the worker processes it would automatically respawn, manage worker pools etc
[00:10] nag I pass data to C when non-trivial work is required
[00:10] nanodocumet Sounds interesting.
[00:11] nag is nanodocumet's scenario common? PHP all-around?
[00:11] mkoppanen nag: yes, fairly common
[00:11] mkoppanen people have a lot of business logic in php that they want to keep using
[00:11] mkoppanen and in a way it makes sense if you are a php shop
[00:12] nag yep, understandable
[00:12] mkoppanen i've seen people doing forking and process management in php
[00:12] mkoppanen which i think is added complexity and not the most stable approach
[00:12] nag eeeek
[00:13] nanodocumet I like php for being easy and widely used, but could be slow for many tasks.
[00:14] nag what would an alternative look like? a sort of broker that sits between zmq and the main PHP execution and manages units of work (also written in PHP)?
[00:14] mkoppanen nag: yes
[00:14] mkoppanen nag: it would convert zeromq messages to fcgi client requests
[00:14] mkoppanen and connect to php-fpm socket directly
[00:15] mkoppanen php scripts in that case would receive a request that looks like just like any cgi request
[00:16] mkoppanen another option is to add some sort of broker in between that handles persistence in case the php processes die
[00:16] nag I think that would be embraced by your target audience
[00:16] nanodocumet mkoppanen: how can a php framework differentiate that from a client browser request?
[00:16] nag thorough but simple docs would be required to not scare anyone off, but judging by current zmq documentation, that's a non-issue
[00:17] nag :>
[00:17] mkoppanen nanodocumet: there should be no need to
[00:18] nanodocumet I am not an expert on the behind the scenes for php, why there should be no need?
[00:20] mkoppanen nanodocumet: you could use your normal framework code to handle the work units
[00:21] nanodocumet like if the request was made by a browser client right?
[00:22] nag oh, one little niggle I found when getting PHP via apache to talk to our daemons was the user/group of apache must match the daemon for IPC
[00:22] nag I was forced to match the user/group up because chmod 0777 on the ipc file descriptor caused Bad Stuff to happen
[00:23] nag mkoppanen: this may have been because I was using ZMQ_PAIR and IPC - sound familiar?
[00:23] mkoppanen inproc won't work
[00:24] mkoppanen ipc should
[00:24] nag with ZMQ_PAIR?
[00:24] mkoppanen pair is one to one
[00:24] mkoppanen probably not the right pattern if you have potentially many
[00:24] nag it was used one to one
[00:25] mkoppanen but if you are running apache then you might have multiple processes?
[00:25] nag I changed patterns and haven't tried to chmod the ipc fd again
[00:25] nag that's accurate. pardon.
[00:27] mkoppanen the permissions should affect it
[00:27] mkoppanen althought matching group and using 770 might be better
[00:28] nag indeed, thanks
[00:28] nanodocumet mkoppanen and nag: thanks for the info. I will try it out and see which one works better.
[00:37] tkahn hi i'm getting Assertion failed: !pgm_supported mq.cpp:240) on ubuntu 11.04 installed from repos
[00:38] tkahn i wasn't able to find a resolution by googling
[00:38] tkahn this is occuring when using PUSH/PULL
[00:42] mkoppanen hmm
[00:42] mkoppanen which version is that?
[00:43] tkahn let me find out... 1 sec
[00:43] tkahn python-zmq : 2.0.10.1-1
[00:44] mkoppanen what about zeromq itself?
[00:44] mkoppanen 2.0.10?
[00:44] mkoppanen that is age old version
[00:44] mkoppanen there has been literally hundreds of bug fixes after that
[00:44] tkahn k
[00:44] tkahn it seems theres a newer version in 11.04 backports
[00:45] tkahn and i think there's an even newer one in 11.10 which gets release on the 13th
[00:45] tkahn i'll try upgrading
[00:45] mkoppanen 2.1 is the current stable series
[00:46] tkahn thanks
[01:11] tkahn mikko: thanks for your help, installing from source and then from pip was exactly what i needed
[01:17] rgl hi
[01:18] rgl what does it mean "ZMQ_SWAP removed. Writing data to disk should be done on top of 0MQ,
[01:18] rgl on inside it." at https://raw.github.com/zeromq/zeromq3-0/master/NEWS ? does it mean we cannot have persistent queues?
[01:19] taotetek rgl: the ability to swap to disk within the socket has been removed yes
[01:20] rgl how would we implement a persistent queue now? is there some out-of-core project for that?
[01:20] taotetek rgl: the recommendation is that you implement it at a higher level
[01:21] taotetek rgl: I've seen some projects for persistent queue devices I know
[01:21] taotetek rgl: there was one that was python based and used tokyo cabinet for persistent queueing
[01:21] taotetek rgl: where I work we're using rsyslog for persistent queues
[01:22] taotetek rgl: we open sourced our plugins for rsyslog if you could use such a thing
[01:22] rgl hummm, how do you known that a message has leaved the zmq queue?
[01:22] taotetek rgl: https://github.com/aggregateknowledge/rsyslog-zeromq
[01:22] rgl I mean, how do you known its safe to remove the message from persistent storage?
[01:23] taotetek rgl: we support rep output sockets from rsyslog
[01:24] taotetek rgl: so clients request the next message, then upon finishing with the message request the next etc
[01:25] taotetek rgl: it's not 100% air tight, it's conceivable you could lose a message
[01:25] grantr taotetek, sorry for late arrival, just curious - has swap option been removed?
[01:25] taotetek but we're working on improving the plugins
[01:26] taotetek grantr_: it's getting removed in either 3.0 or 4.0 (can't remember which version)
[01:26] grantr k
[01:26] rgl from NEWS its already gone.
[01:27] rgl https://raw.github.com/zeromq/zeromq3-0/master/NEWS
[01:27] rgl taotetek, https://github.com/aggregateknowledge/rsyslog-zeromq still uses SWAP.
[01:28] grantr looking at the docs for socket, it seems like ROUTER and DEALER language has been removed in v3
[01:28] taotetek rgl: we don't use swap
[01:28] grantr are those no longer valid socket types?
[01:28] taotetek rgl: but it supports swap because it's built on 2.1.x
[01:29] taotetek rgl: I don't have a need for it because rsyslog is providing my persistence so having a zeromq swap file would just be redundant
[01:29] rgl taotetek, ah ok. I don't known how rsyslog works.
[01:30] taotetek rgl: it's like syslog, with a router built in
[01:30] taotetek and the ability to configure multiple asyncronous queues
[01:31] taotetek you can do direct connections (no queue at all), in memory queues, disk assisted queues
[01:31] taotetek and "linkedlist" which is what we use
[01:31] taotetek you set an in memory queue high water mark
[01:31] taotetek once that water mark is exceeded it spools to on disk queue
[01:31] grantr taotetek, curious why you would use rsyslog instead of rabbitmq?
[01:31] taotetek grantr_: all the data we process is pretty much log files
[01:31] rgl so the CODESTARTdoAction code is run by rsyslog to tell omzeromq to try to queue the message, if it succeeds, then rsyslog can safely drop the message?
[01:32] rgl that is why you are using zmq_send with the ZMQ_NOBLOCK flag?
[01:32] rgl I'm reading the code at https://github.com/aggregateknowledge/rsyslog-zeromq/blob/master/omzeromq/omzeromq.c
[01:32] taotetek rgl: I believe that's correct
[01:33] taotetek rgl: I'll drag Ken in here sometime when he's around, he's the author of the code
[01:33] taotetek grantr_: rsyslog just seemed like a natural fit
[01:33] taotetek grantr_: we take log data from edge servers and route it to various distributed processing systems
[01:34] grantr taotetek, i steered clear of rsyslog because its not runtime-routable and it would be hard to avoid syslog protocol
[01:34] taotetek between data centers as well
[01:34] grantr but you seem to be replacing syslog protocol with zmq iiuc
[01:34] taotetek grantr_: we use JSON messages
[01:34] taotetek grantr_: sent over syslog protocol yes
[01:34] grantr ah
[01:34] grantr taotetek, syslog protocol is not very reliable though
[01:35] taotetek grantr_: rsyslog supports tcp and RELP
[01:35] grantr are you using relp?
[01:35] grantr i have never heard from anyone who has used it
[01:35] taotetek we're currently using tcp in production and starting to set up relp in our QA environment
[01:36] taotetek tcp syslog has been really reliable in pratice for us
[01:36] taotetek we handle around a billion "events" a day at the moment
[01:41] rgl hey sustrik
[01:47] mikko rgl: swap was never persistent
[01:48] rgl mikko, oh really? my bad...
[01:48] mikko it was just for offloading to disk when enough memory is used
[01:48] mikko which is exactly what your OS does automatically
[01:48] rgl so how would one implement persistency?
[01:49] mikko what sort of persistency?
[01:50] rgl for example, a producer places messages on a queue. those messages can only the dropped when someone acks them.
[01:50] mikko rgl: you would implement that on top of zeromq
[01:50] mikko rgl: PZQ does that sort of thing
[01:50] rgl PZQ?
[01:50] mikko https://github.com/mkoppanen/pzq
[01:50] rgl ah thx!
[01:51] mikko but persistency is very complicated topic
[01:51] mikko what pzq doesnt give you is guaranteed ordering
[01:51] rgl for my use case, I'm fine with that :)
[01:52] mikko pzq sends ack for producer after the item has been stored into durable store
[01:52] mikko and on the other side it only removes message if it's ACKd
[01:53] mikko you can schedule ACK timeout for consumers after which the message is redeliverd (unless ACKd)
[01:53] mikko it's still pretty much in development
[01:53] rgl I see. I'll look at it :)
[01:55] rgl mikko, it uses KYOTOCABINET for the storage?
[01:55] mikko yes
[01:56] rgl what is the monitoring socket?
[01:56] mikko you can dump different statistics
[01:56] mikko like number of redeliveries, syncs, messages in queue etc
[01:57] rgl its a REQ/REP socket?
[01:57] mikko it's ROUTER on server side
[02:00] rgl can it run on windows?
[02:00] mikko most likely not
[02:01] mikko all the dependencies support windows but i've never tested
[02:02] rgl alright :D
[03:49] tkahn is it possible to send a socket over a message?
[03:51] tkahn i'm basically looking to have multiple publishers push onto the same stream
[04:12] tkahn nvm apparently you can bind or connect PUSH/PULL sockets
[04:12] tkahn nice
[06:34] sustrik tkahn: send a socket as a message?
[06:34] sustrik you can do that inside a process
[06:34] sustrik via inproc
[12:23] mikko http://build.zero.mq/
[12:23] mikko getting back up
[12:26] sustrik working!
[12:26] sustrik thanks
[12:27] mikko it's on temp hardware
[16:12] deam what is the maximum message length for ZMQ?
[16:13] cremes deam: http://api.zeromq.org/2-1:zmq-msg-init-size
[16:14] cremes deam: on a 64-bit platform, size_t is usually a 64-bit number
[16:14] deam plenty :)
[16:15] cremes um, yeah... ;)
[16:16] deam thanks
[16:36] Steve-o Anyone see that Windows 7 actually supports interface names?
[16:37] Steve-o s/7/8/
[16:37] Steve-o part of the new standard for fixing names
[16:55] Steve-o "Consistent Device Naming", that's not easy to find on Google
[22:12] jmoiron i have a python program that is daemonizing and one of the processes (seems to be the second child) has an assertion error coming from zmq via pyzmq; what should i do here to prevent that?
[22:12] jmoiron er, first child.
[22:36] mikko jmoiron: do you create the context in the parent?
[22:40] jmoiron mikko: seems like it.. i think it's gone away after i've not done
[22:44] mikko jmoiron: i dont think the context survives a fork
[22:44] jmoiron no doesn't appear to.. okay i've got things cleared up now; i didn't realize that there was one being created, was buried in someone elses' code.. thanks for the pointer
[22:45] mikko although docs seems to state that it's ok at least for c api
[22:45] mikko not sure about python
[22:48] jmoiron http://zguide.zeromq.org/page:all#Getting-the-Context-Right that is a bit weirdly put but i think that's saying you need to init a new one per proc
[22:49] mikko If you're using the fork() system call, each process needs its own context. If you do zmq_init(3) in the main process before calling fork(), the child processes get their own contexts.
[22:49] mikko it seems to indicate that it's ok
[22:49] mikko not sure
[22:50] jmoiron well in any case removing any accidental zmq initialization worked; since it all gets started post-daemonization there's no context issues