Changeset 7321
- Timestamp:
- 02/04/08 12:22:40 (10 months ago)
- Files:
-
- plugins/sfJobQueuePlugin/trunk/README (modified) (21 diffs)
Legend:
- Unmodified
- Added
- Removed
- Modified
- Copied
- Moved
plugins/sfJobQueuePlugin/trunk/README
r5677 r7321 6 6 === What the **** ? === 7 7 This plugins enables job queues into Symfony. It includes all the common job 8 queues tasks (start, stop, scheduling through job election strategies, etc.), 9 command line tasks, and a graphical interface for managing queues and jobs. 10 Using a job queue can be useful when asynchronised server-side operations have 8 queues tasks (start, stop, scheduling through job election strategies, logging, 9 etc.), command line tasks, and a graphical interface for managing queues and 10 jobs. 11 12 Using a job queue can be useful when asynchronous server-side operations have 11 13 to be performed (periodically grabbing a RSS feed, automatically sending 12 emails, etc.) or in environments without a cron access.14 emails, re-sampling a video, etc.) or in environments without a cron access. 13 15 14 16 The advantages compared to cron are: 15 * a more ac urate job-execution scheduling (cron's best precision is at the minutelevel)16 * error logging17 * a more accurate job-execution scheduling (cron's best precision is at the minute level, the sfjobQueuePlugin's one is at the second level) 18 * detailed error logging 17 19 * jobs creation from a graphical interface 18 20 * API to create jobs programmatically … … 21 23 * a shell access 22 24 * the right to run PHP from the command line 23 * this plugin has not been tested with windows 25 * the right to override {{{max_execution_time}}} 26 27 Windows user, the plugin has not been tested with windows. 24 28 25 29 === Glossary === 26 30 A "job" is a task that has to be done. In this plugin, it is modeled as a call 27 to a JobHandler, which understands the parameters associated to the job in28 order to perform a dedicated action.31 to a {{{JobHandler}}}, which understands the parameters associated to the job 32 in order to perform a dedicated action. 29 33 30 34 A "job queue" is a group of jobs to be run, eventually several times, 31 associated to a job election strategy. For the moment, one job queue can only32 run one job at a time.35 associated to a job election strategy. One job queue can only run one job at a 36 time, and there is at the moment no dependency mechanism between jobs. 33 37 34 38 The "job queue manager" is the process that is responsible for the launch of 35 39 the job queues. It creates a PHP process for every queue the user has requested 36 40 the start. 41 42 A "job scheduler" is a class responsible of the election of the next job to be 43 runned, given several election models. 37 44 38 45 == Features == … … 40 47 * one given queue can contain heterogeneous job types 41 48 * scheduling (job election) strategy abstraction 42 * support for rec curing jobs49 * support for recurring jobs 43 50 * CLI tasks 51 * detailed logging capacities 44 52 * Queue and jobs management graphical module: 45 53 * job-queues start/stop 46 54 * job-queues statistics 47 55 * job creation 56 * possibility to run a job manually, directly from the graphical interface 48 57 49 58 === Job-queues admin panel === … … 59 68 }}} 60 69 61 * rebuild the model :70 * rebuild the model, the sql instructions, and load it into the database: 62 71 {{{ 63 72 ./symfony propel-build-all 64 73 }}} 65 74 66 * enable the "sfJob" and the "sfJobQueue" modules in your app's settings.yml :75 * enable the "sfJob" and the "sfJobQueue" modules in your app's settings.yml. As these modules are administration interfaces, you will want to only enable it in your administration application: 67 76 {{{ 68 77 enabled_modules: [default, sfJob, sfJobQueue] … … 78 87 79 88 === Starting the job queues manager === 80 The "queue manager" is the process that is responsible for the management of 81 the job queues. When requested, it is able a given job queue. No job will be run 82 until you start the queues manager. 83 84 As this process must be able to create child processes, it has to be runned from 89 The "job queue manager" is the process that is responsible for the management of 90 the job queues. When requested from the graphical interface, it is able to 91 start a given job queue. No job will be run until you start the job queue 92 manager. 93 94 As this process must be able to create child processes, it has to be run from 85 95 the command line. This notably means that you must have a shell access to your 86 96 server in order to be able to use this plugin. … … 88 98 Starting the queue manager can be done with the following Symfony task: 89 99 {{{ 90 $./symfony sfqueue-start-queuemanager YOUR_APP_NAME100 ./symfony sfqueue-start-queuemanager YOUR_APP_NAME 91 101 }}} 92 102 … … 94 104 your server. Therefore, you should rather run the following command: 95 105 {{{ 96 $ nohup ./symfony sfqueue-start-queuemanager YOUR_APP_NAME & 97 }}} 98 99 Doing so will forbid you to stop it without using the "kill" command. If this 100 sounds like a problem to you, you will probably be interested in the 106 nohup ./symfony sfqueue-start-queuemanager YOUR_APP_NAME >> /tmp/nohup.out & 107 }}} 108 109 The content of the {{{/tmp/nohup.out}}} file will contain basic messages from 110 the job queue manager execution, that might be useful when the job queue 111 manager process unexpectedly dies. 112 113 Using nohup will forbid you to stop the job queue manager using CTRL+C, and you 114 will have to rather use the "kill" command. If this sounds like a problem to 115 you, you will probably be interested in the 101 116 [http://www.gnu.org/software/screen screen] utility. 102 117 … … 123 138 124 139 === Running a job queue === 125 Starting a job queue can either be done from the graphical interface, or directly126 in the code:140 Starting a job queue can either be done from the graphical interface (simply 141 click on the symbol in the "is_running" column), or directly in the code: 127 142 {{{ 128 143 #!php … … 137 152 }}} 138 153 139 So, for instance : 140 154 So, for instance: 141 155 {{{ 142 156 ./symfony sfqueue-start-queue frontend 'RSS grabbing queue' 143 157 }}} 158 159 The {{{sfqueue-start-queue}}} task can be used even if the job queue manager is 160 not launched. This is particularly useful if you don't want to use the job queue 161 manager, but prefer to use cron to periodically start/stop a specific job queue. 162 163 It also means something else: when using the {{{sfqueue-start-queue}}} task, the 164 process that will be created won't be a child of the job queue manager and 165 therefore, if the job queue manager is stopped or killed, the job queues that 166 have been launched with {{{sfqueue-start-queue}}} will continue their execution. 144 167 145 168 === Adding a job to the JobQueue === … … 149 172 #!php 150 173 <?php 151 $queue->addJob('mail', array('to' => 'xavier@lacot.org', 'topic' => 'Testing the MailingJobQueue :)')); 174 // retrieve the job queue 175 $queue = sfJobQueuePeer::retrieveByQueueName('mailings queue'); 176 177 // add a new job of type "mail" in it 178 $queue->addJob('mail', 179 array('to' => 'xavier@lacot.org', 180 'topic' => 'Test of the mailings queue :-)')); 152 181 }}} 153 182 … … 156 185 details, see the paragraph "[#Creatinganewjobtype Creating a new job type]". 157 186 187 === Monitoring the job queues and the job queue manager === 188 Suppose you have launched the job manager, and that it is running since several 189 hours or days. If you want to monitor that it is still running, and did not die 190 since you have launched it, you may want to use the {{{ps}}} UNIX command: 191 {{{ 192 ps aux |grep 'php symfony sfqueue-start-queue YOUR_APP_NAME YOUR_QUEUE_NAME' |wc |awk '{print ($1 >= 2)}' 193 }}} 194 195 So, for instance: 196 {{{ 197 ps aux |grep 'php symfony sfqueue-start-queue backend rss_feeds' |wc |awk '{print ($1 >= 2)}' 198 }}} 199 200 If the process is found, this command will display "1". Else, it will display 201 "0". 202 203 === Stopping the job queue manager === 204 For stopping the job queue manager, you simply have to kill it. However, please 205 take care that all the job queues that have been launched using the graphical 206 interface (and so, that are child-processes of the job queue manager) should be 207 stopped before, using the graphical interface. This will ensure that all the 208 running jobs are completed when killing the job queue manager. 209 210 In order to kill the job queue manager, please run the following commands: 211 {{{ 212 $ ps aux| grep 'php ./symfony sfqueue-start-queuemanager backend' 213 xavier 2691 0.0 -0.5 52940 10156 p1 S 08:04AM 0:00.85 php ./symfony sfqueue-start-queuemanager backend 214 xavier 2732 0.0 -0.0 27368 456 p1 S+ 08:26AM 0:00.00 grep php ./symfony sfqueue-start-queuemanager backend 215 }}} 216 217 The job queue manager process pid is 2691, let's kill it: 218 {{{ 219 $ kill -9 2691 220 }}} 221 158 222 == Job parameters == 159 223 A job queue can be tweaked using several parameters: 160 * scheduler_name: the scheduler name defines the policy of job election. For instance, with a "fifo" scheduler, the oldest eligible job will be the one who will be processed next. With a "priority" scheduler, the eligible job with the highest priority will be the first out.161 * polling_delay: time between the end of the execution of one job, and the next job election224 * {{{scheduler_name}}}: the scheduler name defines the policy of job election. For instance, with a "fifo" scheduler, the oldest eligible job will be the one who will be processed next. With a "priority" scheduler, the eligible job with the highest priority will be the first out. 225 * {{{polling_delay}}}: time between the end of the execution of one job, and the next job election 162 226 163 227 [[Image(sfJobQueuePlugin_job_election.png)]] 164 228 165 229 At the job level, it is also possible to set some general execution parameters: 166 * max_tries: 167 * retry_delay: retry delay, in seconds (minimal delay between two tries of the same job) 168 * priority: priority : from 0 to 9 (lower to higher) 169 * params: an array of parameters for the job execution 170 * scheduled_at: date of scheduling after which the job can be runned 230 * {{{max_tries}}}: the maximal number of tries for a non-recurring job. 231 * {{{is_recurring}}}: if the job is recurring, it will be executed every {{{retry_delay}}} seconds. 232 * {{{retry_delay}}}: retry delay, in seconds (minimal delay between two tries of the same job) 233 * {{{priority}}}: priority, from 0 to 9 (lower to higher) 234 * {{{params}}}: an array of parameters for the job execution 235 * {{{scheduled_at}}}: date of scheduling after which the job can be runned 171 236 172 237 == Creating a new job type == … … 174 239 be the less painful possible for the developer. Actually, developping a new job 175 240 type only requires one single {{{JobHandler}}} class to be written, that 176 implements the interface177 [browser:plugins/sfJobQueuePlugin/ lib/jobHandlers/sfJobHandlerInterface.class.php sfJobHandlerInterface].241 extends the class 242 [browser:plugins/sfJobQueuePlugin/trunk/lib/jobHandlers/sfJobHandler.class.php sfJobHandler]. 178 243 Nothing else. 179 244 … … 188 253 #!php 189 254 <?php 190 class sfMailJobHandler extends sfJobHandler implements sfJobHandlerInterface255 class sfMailJobHandler extends sfJobHandler 191 256 { 192 257 public function getParamFields() … … 195 260 } 196 261 197 public function run( $params)262 public function run(array $params) 198 263 { 199 if (mail($params['to'], 200 $params['subject'], 201 $params['message'], 264 if (mail($params['to'], 265 $params['subject'], 266 $params['message'], 202 267 'From: '.$params['from'])) 203 268 { 269 $this->logger->log(sprintf('{sfMailJobHandler} successfully sent mail %s -> %s', $params['from'], $params['to'])); 204 270 return sfJob::SUCCESS; 205 271 } … … 207 273 { 208 274 throw new Exception('There was an error while sending the mail.'); 209 return sfJob::ERROR;210 275 } 211 276 } … … 222 287 [[Image(sfJobQueuePlugin_job_creation_detail.png)]] 223 288 224 === run( $params) ===289 === run(array $params) === 225 290 The method '''{{{run()}}}''' uses an array of parameters for completing the 226 291 job. It must do a proper use of the return status, either {{{sfJob::SUCCESS}}} … … 233 298 234 299 The {{{run()}}} method may launch exceptions. In this case, the job will be 235 marked as failed, except if it is a recur ing job.300 marked as failed, except if it is a recurring job. 236 301 237 302 === static postSave($job, $params) === … … 242 307 243 308 If the job creation must perform other operations that the job execution must 244 be aware of, then the option nal static method {{{postSave($job, $params)}}} is309 be aware of, then the optional static method {{{postSave($job, $params)}}} is 245 310 the right place for this. 246 311 247 312 For instance: 248 313 {{{ 314 #!php 315 249 316 public static function postSave($job, $params) 250 317 { … … 258 325 }}} 259 326 327 == Logging == 328 The plugin proposes a logging system, that is used along with Symfony's default 329 logging system. This logger uses Propel, and is rather useful when debugging 330 in-development jobHandlers. 331 332 If you believe that the JobHandlers that you developed are robust and don't 333 require such a precise logging, you may want to disable this logging by setting 334 the parameter {{{app_sfJobQueuePlugin_logging_enabled}}} in the app.yml file: 335 {{{ 336 all: 337 sfJobQueuePlugin: 338 logging_enabled: false 339 }}} 340 Enabling this logging is completely independent from Symfony's own logging. If 341 you want to enable logging only for the jobs, then disable Symfony's logging, 342 and leave the sfJobQueuePlugin's one enabled. 343 344 The plugin's logging system allows to add log messages from within a job handler: 345 {{{ 346 #!php 347 348 if ($condition) 349 { 350 $this->logger->log('{sfTestJobHandler} successfully passed condition'); 351 return sfJob::SUCCESS; 352 } 353 else 354 { 355 $this->logger->err('{sfTestJobHandler} i will launch an exception'); 356 throw new Exception('There was an error while sending the mail.'); 357 } 358 }}} 359 360 The class {{{sfPropelJobLogger}}} is compatible with {{{sfLogger}}}, which 361 means that you can use the methods {{{debug($message)}}}, {{{info($message)}}}, 362 {{{notice($message)}}}, {{{warning($message)}}}, {{{err($message)}}}, 363 {{{crit($message)}}}, {{{alert($message)}}}, {{{emerg($message)}}} and 364 {{{log($message, $priority = SF_LOG_INFO)}}}. 365 260 366 == API == 261 367 sfJobQueue: 262 368 * '''{{{addJob($type = '', $options = null)}}}''' - Creates a new job in the queue 263 * '''{{{getNbActiveJobs()}}}''' - Returns the number of active jobs, ie. jobs that are neither failed, nor successful, nor cancel led264 * '''{{{getNbActiveReadyJobs()}}}''' - Returns the number of active jobs ready to be run ned265 * '''{{{getNbActiveRecur ingJobs()}}}''' - Returns the number of active recuring jobs369 * '''{{{getNbActiveJobs()}}}''' - Returns the number of active jobs, ie. jobs that are neither failed, nor successful, nor canceled 370 * '''{{{getNbActiveReadyJobs()}}}''' - Returns the number of active jobs ready to be run 371 * '''{{{getNbActiveRecurringJobs()}}}''' - Returns the number of active recurring jobs 266 372 * '''{{{getNbActiveScheduledJobs()}}}''' - Returns the number of active scheduled jobs (ie. scheduled in the future) 267 * '''{{{getNbActiveWaitingJobs()}}}''' - Returns the number of active waiting jobs (ie. active, but not ready to be run ned)268 * '''{{{getNbCompletedCancelledJobs()}}}''' - Returns the number of jobs that have been cancel led373 * '''{{{getNbActiveWaitingJobs()}}}''' - Returns the number of active waiting jobs (ie. active, but not ready to be run) 374 * '''{{{getNbCompletedCancelledJobs()}}}''' - Returns the number of jobs that have been canceled 269 375 * '''{{{getNbCompletedFailureJobs()}}}''' - Returns the number of jobs completed on failure 270 * '''{{{getNbCompletedSuccessfulJobs()}}}''' - Returns the number of successfully run nedjobs376 * '''{{{getNbCompletedSuccessfulJobs()}}}''' - Returns the number of successfully run jobs 271 377 * '''{{{getNbCompletedJobs()}}}''' - Returns the number of completed jobs 272 378 * '''{{{getStatusText()}}}''' - Returns the status of the queue, as a text … … 278 384 * '''{{{run()}}}''' - Runs the job 279 385 386 == Plugin's configuration == 387 There are several {{{app.yml}}} configuration variables: 388 {{{ 389 all: 390 sfJobQueuePlugin: 391 logfile: /tmp/sfJobQueuePlugin.log # path of the file that logs the activity of the queues started by the job queue manager 392 logging_enabled: false # whether the detailed logging should be enabled, or not 393 max_tries: 3 # default value for max_tries for a new job 394 memory: 512M # max memnory to be used when running a job manually from the graphical interface 395 php: php # path of the php binary to be used. If not set, the plugin will use the value of sfToolkit::getPHPCli() 396 queues_polling: 5 # delay in seconds of sleep time between two verifications if queues must be started by the job queue manager 397 }}} 398 280 399 == Unit testing == 281 400 To be done. 282 401 283 402 == Roadmap == 284 * job messages detail285 * have several jobs running at one time286 403 * unit-testing 287 404 * job reporting hooks (measure jobs progress) 288 * handle depend ancy between jobs405 * handle dependency between jobs 289 406 290 407 Long term runners might want to see: … … 303 420 == Changelog == 304 421 422 === version 0.3 - 2008-02-04 === 423 * introduced sfPropelJobLogger, for a better job activity logging 424 * upgraded the graphical interface: 425 * possibility to manually launch jobs 426 * added display logging messages for the last job execution 427 * improved documentation 428 * improved job queue manager robustness 429 305 430 === version 0.2 - 2007-10-23 === 306 * upgraded the graphical interface 431 * upgraded the graphical interface: 307 432 * possibility to create jobs 308 * support for recur ing jobs433 * support for recurring jobs 309 434 * support for jobs scheduling 310 435 * improved documentation
