Clicky

PPSS- a handy tool for parallel processing | PromptCloud
 

PPSS- a handy tool for parallel processing

PPSS- a handy tool for parallel processing

We were looking for a light Linux shell tool, that could process given commands using advantage of multicore system (parallel processing). Two of the tools that we came across is worth writing about – parallel and PPSS. This blog discusses PPSS, while parallel can be food for a future blog.

PPSS can be downloaded from http://code.google.com/p/ppss/ (there are deb and rpm files too). PPSS is a shell script that can be used to run any command, script or program in parallel. All it needs is source (file or dir) and a command to execute.

Example-.

./ppss -d -c ” -p

./ppss -f -c ” -p

If source is a directory, then it executes the command on each file in the directory and if source is a file, it executes the command on each line in the file.

Note: command should always be enclosed in single quotes. Argument (each file in source directory or each line in source file) can be accessed in command by the variable – “$ITEM”. At any time, number of items being processed will never increase the cores available, say while processing 50GB of data.

$~/bin/ppss -f list_of_files_to_be_processed.txt -c ‘zgrep “”

../../”$ITEM”‘ -p 2

Jul 26 09:11:22:  =========================================================                                                                                             

Jul 26 09:11:22:                         |P|P|S|S|

Jul 26 09:11:22:  Distributed Parallel Processing Shell Script vers. 2.97                                                                                               

Jul 26 09:11:22:  =========================================================

Jul 26 09:11:22:  Hostname:             domU-12-31-39-00-EC-96                                                                                                          

Jul 26 09:11:22:  ———————————————————

Jul 26 09:11:22:  CPU: Dual-Core AMD Opteron(tm) Processor 2218 HE                                                                                                      

Jul 26 09:11:22:  Starting 2 parallel workers.

Jul 26 09:11:22:  ———————————————————

Jul 26 13:18:18:  70% complete. Processed 10199 of 14400. Failed 158/14400

Jul 26 13:16:33:  ETA: Thu Jul 26 13:11:23 UTC 2012

Output of each command executed on each item is logged in a single file named after the item. The log file is available at location – ‘./ppss_dir/job_log/’. Command execution status for the item can be obtained from log file and it displays as- “Status: FAILURE / Status: Success “

For more information on ppss, follow the links below.

http://code.google.com/p/ppss/

http://code.google.com/p/ppss/downloads/detail?name=ppss-2.85.tgz

Related Posts

No Comments

Post A Comment

Ready to discuss your requirements?

REQUEST A QUOTE
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.

Price Calculator

  • Total number of websites
  • number of records
  • including one time setup fee
  • from second month onwards
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.
  • This field is for validation purposes and should be left unchanged.