[Time] Name | Message |
[00:00] mkoppanen
|
rather than immediate
|
[00:00] mkoppanen
|
or maybe outbound mail servers are going through maintenance or something similar
|
[00:00] nanodocumet
|
but the message will be queue right?
|
[00:00] mkoppanen
|
yes
|
[00:01] mkoppanen
|
it all depends on how you handle it
|
[00:01] mkoppanen
|
which language are you using?
|
[00:01] nanodocumet
|
PHP.
|
[00:01] mkoppanen
|
for the worker processes as wlel?
|
[00:01] nanodocumet
|
I have installed zeromq and php bindings on ubuntu.
|
[00:01] mkoppanen
|
as well*
|
[00:02] nanodocumet
|
yes, I am planning on using PHP for all.
|
[00:02] nanodocumet
|
I use Kohana framework.
|
[00:02] nanodocumet
|
(that's where I heard about zeromq).
|
[00:03] nanodocumet
|
I already have some scripts (running using crontab) to send the emails.
|
[00:04] nanodocumet
|
I was planning on adding the messages (tasks for sending the emails with all the information) from Kohana.
|
[00:04] mkoppanen
|
i guess samsoir is the champion of zeromq in the kohana community
|
[00:04] nanodocumet
|
yes!
|
[00:04] nanodocumet
|
he mentioned it!
|
[00:05] mkoppanen
|
i used to work with him back in the day
|
[00:05] mkoppanen
|
in london
|
[00:05] mkoppanen
|
you could start testing with simple PUSH/PULL pattern
|
[00:05] nag
|
nanodocumet: fyi, the PHP implementation has worked well for me. we pass data between a set of C daemons and PHP via zmq msgs.
|
[00:05] mkoppanen
|
and extend from there to suit the specific scenario
|
[00:06] mkoppanen
|
nag: good to hear
|
[00:06] nanodocumet
|
nag: that's what I've been reading, and what I really like.
|
[00:06] nag
|
that first moment when things 'just work' is refreshing
|
[00:07] mkoppanen
|
so, now that we have some php users here i got a question for you guys
|
[00:07] mkoppanen
|
would you find useful if there was a tool that allowed communication with zmq
|
[00:07] mkoppanen
|
and would manage worker processes over fcgi
|
[00:07] mkoppanen
|
so that php-fpm could be used to manage workers
|
[00:07] mkoppanen
|
and the requests would look like just like any other fastcgi request
|
[00:09] nanodocumet
|
I think that's what samsoir mentioned.
|
[00:09] nag
|
mkoppanen: I believe so. Does the PHP implementation not work with something like nginx then?
|
[00:09] mkoppanen
|
nag: it does
|
[00:09] mkoppanen
|
nag: but what i see often that people write worker processes in php
|
[00:09] mkoppanen
|
which is not ideal but probably works
|
[00:10] nag
|
ah, I see... I wouldn't personally find it useful in my current work
|
[00:10] mkoppanen
|
if you could use php-fpm to manage the worker processes it would automatically respawn, manage worker pools etc
|
[00:10] nag
|
I pass data to C when non-trivial work is required
|
[00:10] nanodocumet
|
Sounds interesting.
|
[00:11] nag
|
is nanodocumet's scenario common? PHP all-around?
|
[00:11] mkoppanen
|
nag: yes, fairly common
|
[00:11] mkoppanen
|
people have a lot of business logic in php that they want to keep using
|
[00:11] mkoppanen
|
and in a way it makes sense if you are a php shop
|
[00:12] nag
|
yep, understandable
|
[00:12] mkoppanen
|
i've seen people doing forking and process management in php
|
[00:12] mkoppanen
|
which i think is added complexity and not the most stable approach
|
[00:12] nag
|
eeeek
|
[00:13] nanodocumet
|
I like php for being easy and widely used, but could be slow for many tasks.
|
[00:14] nag
|
what would an alternative look like? a sort of broker that sits between zmq and the main PHP execution and manages units of work (also written in PHP)?
|
[00:14] mkoppanen
|
nag: yes
|
[00:14] mkoppanen
|
nag: it would convert zeromq messages to fcgi client requests
|
[00:14] mkoppanen
|
and connect to php-fpm socket directly
|
[00:15] mkoppanen
|
php scripts in that case would receive a request that looks like just like any cgi request
|
[00:16] mkoppanen
|
another option is to add some sort of broker in between that handles persistence in case the php processes die
|
[00:16] nag
|
I think that would be embraced by your target audience
|
[00:16] nanodocumet
|
mkoppanen: how can a php framework differentiate that from a client browser request?
|
[00:16] nag
|
thorough but simple docs would be required to not scare anyone off, but judging by current zmq documentation, that's a non-issue
|
[00:17] nag
|
:>
|
[00:17] mkoppanen
|
nanodocumet: there should be no need to
|
[00:18] nanodocumet
|
I am not an expert on the behind the scenes for php, why there should be no need?
|
[00:20] mkoppanen
|
nanodocumet: you could use your normal framework code to handle the work units
|
[00:21] nanodocumet
|
like if the request was made by a browser client right?
|
[00:22] nag
|
oh, one little niggle I found when getting PHP via apache to talk to our daemons was the user/group of apache must match the daemon for IPC
|
[00:22] nag
|
I was forced to match the user/group up because chmod 0777 on the ipc file descriptor caused Bad Stuff to happen
|
[00:23] nag
|
mkoppanen: this may have been because I was using ZMQ_PAIR and IPC - sound familiar?
|
[00:23] mkoppanen
|
inproc won't work
|
[00:24] mkoppanen
|
ipc should
|
[00:24] nag
|
with ZMQ_PAIR?
|
[00:24] mkoppanen
|
pair is one to one
|
[00:24] mkoppanen
|
probably not the right pattern if you have potentially many
|
[00:24] nag
|
it was used one to one
|
[00:25] mkoppanen
|
but if you are running apache then you might have multiple processes?
|
[00:25] nag
|
I changed patterns and haven't tried to chmod the ipc fd again
|
[00:25] nag
|
that's accurate. pardon.
|
[00:27] mkoppanen
|
the permissions should affect it
|
[00:27] mkoppanen
|
althought matching group and using 770 might be better
|
[00:28] nag
|
indeed, thanks
|
[00:28] nanodocumet
|
mkoppanen and nag: thanks for the info. I will try it out and see which one works better.
|
[00:37] tkahn
|
hi i'm getting Assertion failed: !pgm_supported mq.cpp:240) on ubuntu 11.04 installed from repos
|
[00:38] tkahn
|
i wasn't able to find a resolution by googling
|
[00:38] tkahn
|
this is occuring when using PUSH/PULL
|
[00:42] mkoppanen
|
hmm
|
[00:42] mkoppanen
|
which version is that?
|
[00:43] tkahn
|
let me find out... 1 sec
|
[00:43] tkahn
|
python-zmq : 2.0.10.1-1
|
[00:44] mkoppanen
|
what about zeromq itself?
|
[00:44] mkoppanen
|
2.0.10?
|
[00:44] mkoppanen
|
that is age old version
|
[00:44] mkoppanen
|
there has been literally hundreds of bug fixes after that
|
[00:44] tkahn
|
k
|
[00:44] tkahn
|
it seems theres a newer version in 11.04 backports
|
[00:45] tkahn
|
and i think there's an even newer one in 11.10 which gets release on the 13th
|
[00:45] tkahn
|
i'll try upgrading
|
[00:45] mkoppanen
|
2.1 is the current stable series
|
[00:46] tkahn
|
thanks
|
[01:11] tkahn
|
mikko: thanks for your help, installing from source and then from pip was exactly what i needed
|
[01:17] rgl
|
hi
|
[01:18] rgl
|
what does it mean "ZMQ_SWAP removed. Writing data to disk should be done on top of 0MQ,
|
[01:18] rgl
|
on inside it." at https://raw.github.com/zeromq/zeromq3-0/master/NEWS ? does it mean we cannot have persistent queues?
|
[01:19] taotetek
|
rgl: the ability to swap to disk within the socket has been removed yes
|
[01:20] rgl
|
how would we implement a persistent queue now? is there some out-of-core project for that?
|
[01:20] taotetek
|
rgl: the recommendation is that you implement it at a higher level
|
[01:21] taotetek
|
rgl: I've seen some projects for persistent queue devices I know
|
[01:21] taotetek
|
rgl: there was one that was python based and used tokyo cabinet for persistent queueing
|
[01:21] taotetek
|
rgl: where I work we're using rsyslog for persistent queues
|
[01:22] taotetek
|
rgl: we open sourced our plugins for rsyslog if you could use such a thing
|
[01:22] rgl
|
hummm, how do you known that a message has leaved the zmq queue?
|
[01:22] taotetek
|
rgl: https://github.com/aggregateknowledge/rsyslog-zeromq
|
[01:22] rgl
|
I mean, how do you known its safe to remove the message from persistent storage?
|
[01:23] taotetek
|
rgl: we support rep output sockets from rsyslog
|
[01:24] taotetek
|
rgl: so clients request the next message, then upon finishing with the message request the next etc
|
[01:25] taotetek
|
rgl: it's not 100% air tight, it's conceivable you could lose a message
|
[01:25] grantr
|
taotetek, sorry for late arrival, just curious - has swap option been removed?
|
[01:25] taotetek
|
but we're working on improving the plugins
|
[01:26] taotetek
|
grantr_: it's getting removed in either 3.0 or 4.0 (can't remember which version)
|
[01:26] grantr
|
k
|
[01:26] rgl
|
from NEWS its already gone.
|
[01:27] rgl
|
https://raw.github.com/zeromq/zeromq3-0/master/NEWS
|
[01:27] rgl
|
taotetek, https://github.com/aggregateknowledge/rsyslog-zeromq still uses SWAP.
|
[01:28] grantr
|
looking at the docs for socket, it seems like ROUTER and DEALER language has been removed in v3
|
[01:28] taotetek
|
rgl: we don't use swap
|
[01:28] grantr
|
are those no longer valid socket types?
|
[01:28] taotetek
|
rgl: but it supports swap because it's built on 2.1.x
|
[01:29] taotetek
|
rgl: I don't have a need for it because rsyslog is providing my persistence so having a zeromq swap file would just be redundant
|
[01:29] rgl
|
taotetek, ah ok. I don't known how rsyslog works.
|
[01:30] taotetek
|
rgl: it's like syslog, with a router built in
|
[01:30] taotetek
|
and the ability to configure multiple asyncronous queues
|
[01:31] taotetek
|
you can do direct connections (no queue at all), in memory queues, disk assisted queues
|
[01:31] taotetek
|
and "linkedlist" which is what we use
|
[01:31] taotetek
|
you set an in memory queue high water mark
|
[01:31] taotetek
|
once that water mark is exceeded it spools to on disk queue
|
[01:31] grantr
|
taotetek, curious why you would use rsyslog instead of rabbitmq?
|
[01:31] taotetek
|
grantr_: all the data we process is pretty much log files
|
[01:31] rgl
|
so the CODESTARTdoAction code is run by rsyslog to tell omzeromq to try to queue the message, if it succeeds, then rsyslog can safely drop the message?
|
[01:32] rgl
|
that is why you are using zmq_send with the ZMQ_NOBLOCK flag?
|
[01:32] rgl
|
I'm reading the code at https://github.com/aggregateknowledge/rsyslog-zeromq/blob/master/omzeromq/omzeromq.c
|
[01:32] taotetek
|
rgl: I believe that's correct
|
[01:33] taotetek
|
rgl: I'll drag Ken in here sometime when he's around, he's the author of the code
|
[01:33] taotetek
|
grantr_: rsyslog just seemed like a natural fit
|
[01:33] taotetek
|
grantr_: we take log data from edge servers and route it to various distributed processing systems
|
[01:34] grantr
|
taotetek, i steered clear of rsyslog because its not runtime-routable and it would be hard to avoid syslog protocol
|
[01:34] taotetek
|
between data centers as well
|
[01:34] grantr
|
but you seem to be replacing syslog protocol with zmq iiuc
|
[01:34] taotetek
|
grantr_: we use JSON messages
|
[01:34] taotetek
|
grantr_: sent over syslog protocol yes
|
[01:34] grantr
|
ah
|
[01:34] grantr
|
taotetek, syslog protocol is not very reliable though
|
[01:35] taotetek
|
grantr_: rsyslog supports tcp and RELP
|
[01:35] grantr
|
are you using relp?
|
[01:35] grantr
|
i have never heard from anyone who has used it
|
[01:35] taotetek
|
we're currently using tcp in production and starting to set up relp in our QA environment
|
[01:36] taotetek
|
tcp syslog has been really reliable in pratice for us
|
[01:36] taotetek
|
we handle around a billion "events" a day at the moment
|
[01:41] rgl
|
hey sustrik
|
[01:47] mikko
|
rgl: swap was never persistent
|
[01:48] rgl
|
mikko, oh really? my bad...
|
[01:48] mikko
|
it was just for offloading to disk when enough memory is used
|
[01:48] mikko
|
which is exactly what your OS does automatically
|
[01:48] rgl
|
so how would one implement persistency?
|
[01:49] mikko
|
what sort of persistency?
|
[01:50] rgl
|
for example, a producer places messages on a queue. those messages can only the dropped when someone acks them.
|
[01:50] mikko
|
rgl: you would implement that on top of zeromq
|
[01:50] mikko
|
rgl: PZQ does that sort of thing
|
[01:50] rgl
|
PZQ?
|
[01:50] mikko
|
https://github.com/mkoppanen/pzq
|
[01:50] rgl
|
ah thx!
|
[01:51] mikko
|
but persistency is very complicated topic
|
[01:51] mikko
|
what pzq doesnt give you is guaranteed ordering
|
[01:51] rgl
|
for my use case, I'm fine with that :)
|
[01:52] mikko
|
pzq sends ack for producer after the item has been stored into durable store
|
[01:52] mikko
|
and on the other side it only removes message if it's ACKd
|
[01:53] mikko
|
you can schedule ACK timeout for consumers after which the message is redeliverd (unless ACKd)
|
[01:53] mikko
|
it's still pretty much in development
|
[01:53] rgl
|
I see. I'll look at it :)
|
[01:55] rgl
|
mikko, it uses KYOTOCABINET for the storage?
|
[01:55] mikko
|
yes
|
[01:56] rgl
|
what is the monitoring socket?
|
[01:56] mikko
|
you can dump different statistics
|
[01:56] mikko
|
like number of redeliveries, syncs, messages in queue etc
|
[01:57] rgl
|
its a REQ/REP socket?
|
[01:57] mikko
|
it's ROUTER on server side
|
[02:00] rgl
|
can it run on windows?
|
[02:00] mikko
|
most likely not
|
[02:01] mikko
|
all the dependencies support windows but i've never tested
|
[02:02] rgl
|
alright :D
|
[03:49] tkahn
|
is it possible to send a socket over a message?
|
[03:51] tkahn
|
i'm basically looking to have multiple publishers push onto the same stream
|
[04:12] tkahn
|
nvm apparently you can bind or connect PUSH/PULL sockets
|
[04:12] tkahn
|
nice
|
[06:34] sustrik
|
tkahn: send a socket as a message?
|
[06:34] sustrik
|
you can do that inside a process
|
[06:34] sustrik
|
via inproc
|
[12:23] mikko
|
http://build.zero.mq/
|
[12:23] mikko
|
getting back up
|
[12:26] sustrik
|
working!
|
[12:26] sustrik
|
thanks
|
[12:27] mikko
|
it's on temp hardware
|
[16:12] deam
|
what is the maximum message length for ZMQ?
|
[16:13] cremes
|
deam: http://api.zeromq.org/2-1:zmq-msg-init-size
|
[16:14] cremes
|
deam: on a 64-bit platform, size_t is usually a 64-bit number
|
[16:14] deam
|
plenty :)
|
[16:15] cremes
|
um, yeah... ;)
|
[16:16] deam
|
thanks
|
[16:36] Steve-o
|
Anyone see that Windows 7 actually supports interface names?
|
[16:37] Steve-o
|
s/7/8/
|
[16:37] Steve-o
|
part of the new standard for fixing names
|
[16:55] Steve-o
|
"Consistent Device Naming", that's not easy to find on Google
|
[22:12] jmoiron
|
i have a python program that is daemonizing and one of the processes (seems to be the second child) has an assertion error coming from zmq via pyzmq; what should i do here to prevent that?
|
[22:12] jmoiron
|
er, first child.
|
[22:36] mikko
|
jmoiron: do you create the context in the parent?
|
[22:40] jmoiron
|
mikko: seems like it.. i think it's gone away after i've not done
|
[22:44] mikko
|
jmoiron: i dont think the context survives a fork
|
[22:44] jmoiron
|
no doesn't appear to.. okay i've got things cleared up now; i didn't realize that there was one being created, was buried in someone elses' code.. thanks for the pointer
|
[22:45] mikko
|
although docs seems to state that it's ok at least for c api
|
[22:45] mikko
|
not sure about python
|
[22:48] jmoiron
|
http://zguide.zeromq.org/page:all#Getting-the-Context-Right that is a bit weirdly put but i think that's saying you need to init a new one per proc
|
[22:49] mikko
|
If you're using the fork() system call, each process needs its own context. If you do zmq_init(3) in the main process before calling fork(), the child processes get their own contexts.
|
[22:49] mikko
|
it seems to indicate that it's ok
|
[22:49] mikko
|
not sure
|
[22:50] jmoiron
|
well in any case removing any accidental zmq initialization worked; since it all gets started post-daemonization there's no context issues
|