Difference between revisions of "PM Software Stack"

From Pulsed Media Wiki
Line 19: Line 19:
  
 
== Changelog ==
 
== Changelog ==
=== 15/12/2011 ===
+
=== Changes 2012 ===
 +
==== 28/01/2012 ====
 +
* Added iotop and cfv to be installed from apt repo.
 +
 
 +
=== Changes 2011 ===
 +
 
 +
==== 15/12/2011 ====
 
* Added '''.irssi/config''' to user dirs, autojoining to #PulsedMedia on Freenode when irssi is launched
 
* Added '''.irssi/config''' to user dirs, autojoining to #PulsedMedia on Freenode when irssi is launched
 
* Updated GUI IRC links, Chat tab and Adobe download links, with other small additional information and updates.
 
* Updated GUI IRC links, Chat tab and Adobe download links, with other small additional information and updates.
Line 26: Line 32:
 
* Update-step2 sees if .irssi/config exists and if it does not copies it there
 
* Update-step2 sees if .irssi/config exists and if it does not copies it there
  
=== 30/11/2011 ===
+
==== 30/11/2011 ====
 
* Chosen rTorrent SCGI port range increased from 4k to 8k.
 
* Chosen rTorrent SCGI port range increased from 4k to 8k.
 
* "fractional blocks" for config. Resource calcs are based on number of blocks, and fractions were not allowed prior.
 
* "fractional blocks" for config. Resource calcs are based on number of blocks, and fractions were not allowed prior.
  
=== 29/11/2011 ===
+
==== 29/11/2011 ====
 
* During update also update user crontabs. Added to both update.php and step2, to be removed from step2 after this update has seen significant adoption.
 
* During update also update user crontabs. Added to both update.php and step2, to be removed from step2 after this update has seen significant adoption.
 
* Restarting lighttpd during update, also done doubly for now
 
* Restarting lighttpd during update, also done doubly for now
Line 38: Line 44:
 
* Updated root  & user crontabs to reflect updateQuotas
 
* Updated root  & user crontabs to reflect updateQuotas
  
=== 28/11/2011 ===
+
==== 28/11/2011 ====
 
* quotaFix now removes aquota*new files to avoid collision and thus failed update via quotacheck
 
* quotaFix now removes aquota*new files to avoid collision and thus failed update via quotacheck
 
* userTorrents.php : Lists the count of torrents each user has, for quicker analysis usage stats
 
* userTorrents.php : Lists the count of torrents each user has, for quicker analysis usage stats
  
=== 23/11/2011 ===
+
==== 23/11/2011 ====
 
* Permissions setup script added, which is utilized by recreateUser.
 
* Permissions setup script added, which is utilized by recreateUser.
 
* recreateUser.php fixes
 
* recreateUser.php fixes
  
=== 22/11/2011 ===
+
==== 22/11/2011 ====
 
* Created recreateUser.php which recreates home dir, preserving data+session if existing. ''ALPHA''
 
* Created recreateUser.php which recreates home dir, preserving data+session if existing. ''ALPHA''
  
=== 19/11/2011 ===
+
==== 19/11/2011 ====
 
* Fixed saving API Key.
 
* Fixed saving API Key.
  
=== 17/11/2011 ===
+
==== 17/11/2011 ====
 
* Suspend and Unsuspend scripts fixed
 
* Suspend and Unsuspend scripts fixed
 
* recreateLighttpd backup fixed
 
* recreateLighttpd backup fixed
Line 62: Line 68:
 
* checkInstances now also checks rtorrent configs, and if changed lists them on /root/changedConfigs for admin review
 
* checkInstances now also checks rtorrent configs, and if changed lists them on /root/changedConfigs for admin review
  
=== 25/8/2011 ===
+
==== 25/8/2011 ====
 
Streamlined installer for faster setups :) First release of API, just one command now. Some things moved to BETA status and considered feature complete. Many bug fixes, additional commenting etc. also added
 
Streamlined installer for faster setups :) First release of API, just one command now. Some things moved to BETA status and considered feature complete. Many bug fixes, additional commenting etc. also added
  
Line 90: Line 96:
  
  
=== 8/6/2011 ===
+
==== 8/6/2011 ====
 
Bug fixes.
 
Bug fixes.
  
Line 113: Line 119:
 
* Debian 5 add cron jobs to root.cron.additional config, create if needed
 
* Debian 5 add cron jobs to root.cron.additional config, create if needed
 
    
 
    
=== 31/5/2011 ===
+
==== 31/5/2011 ====
  
 
* GUI welcome page checking if proper quota information was passed has been added.
 
* GUI welcome page checking if proper quota information was passed has been added.
Line 120: Line 126:
 
* GUI welcome page quota text styles: White and location
 
* GUI welcome page quota text styles: White and location
  
=== 30/5/2011 ===
+
==== 30/5/2011 ====
  
 
* GUI update script quota checking added (doesn't try to update when user is over quota burst limit, no more blanking GUI when update happens!)
 
* GUI update script quota checking added (doesn't try to update when user is over quota burst limit, no more blanking GUI when update happens!)
Line 131: Line 137:
 
* Basic [[Idempotency|idempotent]] lighttpd configurator (alpha, not suitable for production, TESTING only)
 
* Basic [[Idempotency|idempotent]] lighttpd configurator (alpha, not suitable for production, TESTING only)
  
=== 23/5/2011 ===
+
==== 23/5/2011 ====
  
 
* terminateUser.php http password deletion fix, text fixes, executability.
 
* terminateUser.php http password deletion fix, text fixes, executability.
Line 138: Line 144:
 
* crons: increased check http to 4 minute intervals (avoid http timeout period issues)
 
* crons: increased check http to 4 minute intervals (avoid http timeout period issues)
  
=== 18/5/2011 ===
+
==== 18/5/2011 ====
 
Temporary fix to bypass userDb until fully utilized. You may use this update or use /scripts/util/populateUserDb.php to fix the issue.
 
Temporary fix to bypass userDb until fully utilized. You may use this update or use /scripts/util/populateUserDb.php to fix the issue.
  
 
* Changed listUsers.php to always use systemUsers.php, as fixing userDb saving introduced a bug where ie. checkInstances was not working fully (using userDb, which lacks most users in most servers)
 
* Changed listUsers.php to always use systemUsers.php, as fixing userDb saving introduced a bug where ie. checkInstances was not working fully (using userDb, which lacks most users in most servers)
  
=== 16/5/2011 ===
+
==== 16/5/2011 ====
  
 
Few bug fixes and minor features introduced.
 
Few bug fixes and minor features introduced.
Line 162: Line 168:
  
  
=== 15/5/2011 ===
+
==== 15/5/2011 ====
  
 
Couple important new features introduced, several bug fixes.
 
Couple important new features introduced, several bug fixes.
Line 183: Line 189:
  
  
=== 27/03/2011 ===
+
==== 27/03/2011 ====
  
 
This is primarily a bug fix and code quality release.
 
This is primarily a bug fix and code quality release.

Revision as of 14:29, 28 January 2012

PMSS, short of PM Software Stack is a compilation of scripts to manage single server side seedbox configuration, turning instantly and conveniently an regular Debian 5.0 server into a seedbox. The PMSS does not contain multiserver management utilities nor clustering support out-of-box, but does include PM Master GUI remote update features. Debian 6.0 squeeze support is still in unstable status but several production servers already do exist. PMSS currently only supports rTorrent+ruTorrent stack.

General features


Using PMSS

See Category:PM Software Stack Guides

Changelog

Changes 2012

28/01/2012

  • Added iotop and cfv to be installed from apt repo.

Changes 2011

15/12/2011

  • Added .irssi/config to user dirs, autojoining to #PulsedMedia on Freenode when irssi is launched
  • Updated GUI IRC links, Chat tab and Adobe download links, with other small additional information and updates.
  • Removed Network Storage tab from GUI because Dropbox is blocking pageviews coming from our system for unknown reason.
  • Added Proxy information on GUI
  • Update-step2 sees if .irssi/config exists and if it does not copies it there

30/11/2011

  • Chosen rTorrent SCGI port range increased from 4k to 8k.
  • "fractional blocks" for config. Resource calcs are based on number of blocks, and fractions were not allowed prior.

29/11/2011

  • During update also update user crontabs. Added to both update.php and step2, to be removed from step2 after this update has seen significant adoption.
  • Restarting lighttpd during update, also done doubly for now
  • Moved setup api key to regular update script. In several months it will potentially be removed from update all together as it's a installation step, and included to update older nodes.
  • Scripts permissions update
  • updateQuotas.php -> cron to update all user quotas, as discreet user from central cron to ensure it gets updated.
  • Updated root & user crontabs to reflect updateQuotas

28/11/2011

  • quotaFix now removes aquota*new files to avoid collision and thus failed update via quotacheck
  • userTorrents.php : Lists the count of torrents each user has, for quicker analysis usage stats

23/11/2011

  • Permissions setup script added, which is utilized by recreateUser.
  • recreateUser.php fixes

22/11/2011

  • Created recreateUser.php which recreates home dir, preserving data+session if existing. ALPHA

19/11/2011

  • Fixed saving API Key.

17/11/2011

  • Suspend and Unsuspend scripts fixed
  • recreateLighttpd backup fixed
  • Added access format of: servername/user-username/, for example: localhost/user-johndoe/. Thus supporting non FQDN setups
  • addUser.php calls recreateLighttpd instead of just adding the hostname, this ensures lighttpd config is always up-to-date.
  • Added to both update steps recreateLighttpd. It will remain on step2 for the next ~6months to ensure every system out there will have the update update.php
  • Removed the default "testing" torrents
  • Upload slots global, by default 6 times the non-global upload slots. It is "forcefully" inserted even if template does not it have for the moment. this forcefull insert will be removed in the future.
  • checkInstances now also checks rtorrent configs, and if changed lists them on /root/changedConfigs for admin review

25/8/2011

Streamlined installer for faster setups :) First release of API, just one command now. Some things moved to BETA status and considered feature complete. Many bug fixes, additional commenting etc. also added

  • fixed userTransfer chowning lacking -R, thus UID was transferred for many files from the other server causing incorrect permissions
  • wrapped vnstat install into if !file_exists to see if vnstat is installed already.
  • added username 'srvadmin' to filter list on systemUsers
  • removed from installer editing munin.conf -> if used can be done manually. Interim solution.
  • put install vim as first thing on installer
  • setting node hostname straight after installing vim
  • moved setting up lighttpd as last thing
  • scripts, skel, etc are installed earlier in installation
  • SSL cert + quota as last items on setup
  • Executing update-step2 on installer (so you don't need to execute updater after setup to get everything in)
  • /scripts/util/setupRootCron.php -> creating root conf moved out of update.php
  • setupSkelPermissions.php, same reason as root cron setup
  • added setupRootCron and setupSkelPermissions to installer
  • Idempotent lighttpd configurator feature completed. Backups the config first.
  • Terminate user moved to beta status, as using lighttpd configurator now and restarting lighttpd.
  • Added for terminate second argument, if 2nd argument is --confirm, straight accepts termination without asking for approval, for making automation easier.
  • checkHttpd timer increased to 3 seconds before restarting lighttpd
  • terminateUser timer increased to 5 seconds after trying to kill all user processes
  • setupApiKey -> creates node key for API. Invoked upon setup.
  • added authorization key
  • removed /var/www/index.lighttpd.html
  • added /var/www/index.php redir to pulsedmedia.com
  • added beginnings of remote API for easier remote control management, by default in debug mode for now


8/6/2011

Bug fixes.

terminateUser.php:

  • Was not actually completing shell userdeletion
  • Added killing user processes (forgotten)
  • added sleeping 2 seconds to allow time for rTorrent/other processes by user to die
  • Added 2nd attempt to userdel after data removal
  • Fixed some typos in texts
  • Added check for user actually exists

NOTE: Still Alpha, non feature complete and not recommended for production use.

  • checkHttpd.php added sleep between lighttpd kill and restart, to avoid PID file mismatch scenario

update-step2.php:

  • First time being used
  • update.php moved cron & ftp config stuff AFTER step2
  • Install vnstat + vim (vim not by default on Debian 6)
  • Setup interface for vnstat
  • Configure vnstat for Debian 6
  • Make vnstat cron jobs for Debian 5: twice a minute update (thus no support for 1GigE interface)
  • Debian 5 add cron jobs to root.cron.additional config, create if needed

31/5/2011

  • GUI welcome page checking if proper quota information was passed has been added.
  • GUI welcome page malformed serialized data fix
  • GUI welcome page quota percentage missing % char
  • GUI welcome page quota text styles: White and location

30/5/2011

  • GUI update script quota checking added (doesn't try to update when user is over quota burst limit, no more blanking GUI when update happens!)
  • Users default quota information update rate has been increased to 10mins
  • HDD Quota plugin commenting added
  • HDD Quota plugin parser bug with bursted amounts fixed
  • HDD Quota now shows USED/QUOTA instead of FREE/QUOTA. Makes more sense.
  • HDD Quota "title" fixed (showing quota info) to be whole area of the meter
  • Welcome page quota meter added :) Basic graph, larger and constantly visible text for quota info :)
  • Basic idempotent lighttpd configurator (alpha, not suitable for production, TESTING only)

23/5/2011

  • terminateUser.php http password deletion fix, text fixes, executability.
  • checkHttpd.php output lighttpd output, killall -9 lighttpd to ensure shutdown, http fetch mute removed so error gets logged
  • update.php: removed apt updates because soft.sh already does this
  • crons: increased check http to 4 minute intervals (avoid http timeout period issues)

18/5/2011

Temporary fix to bypass userDb until fully utilized. You may use this update or use /scripts/util/populateUserDb.php to fix the issue.

  • Changed listUsers.php to always use systemUsers.php, as fixing userDb saving introduced a bug where ie. checkInstances was not working fully (using userDb, which lacks most users in most servers)

16/5/2011

Few bug fixes and minor features introduced.

  • Update script order of items.
  • Update script apt-get options for additional package installation lacked auto confirm.
  • Update script support for "step 2", /scripts/util/update-step2.php is included if present, thus supporting dynamic updates
  • Update script execute binary line added, so it can actually be executed simply by ./update.php
  • /scripts permission fix, only root should have anykind of access (no execute or read for others)
  • Support for root additional items via /etc/seedbox/config/root.cron.additional
  • Add user script minor cleanup, additional commenting.
  • Suspend script comments
  • simple terminate user script, ALPHA -> not feature complete yet.

Updating from 27/03/2011 release or earlier

cd ~; bash soft.sh; php /scripts/update.php

Updating from 15/5/2011 release

php /scripts/update.php


15/5/2011

Couple important new features introduced, several bug fixes.

  • Added lighttpd status checking and auto-restart (very rudimentary)
  • Fixed typo in /scripts/util/userConfig.php
  • Added /etc/seedbox/config/root.cron
  • /scripts/util/quotaFix.php now echoes results (also turns quota off, no need to turn it off first)
  • proftpd config update (remote) with MaxClientsPerUser, MaxClientsPerHost, run /scripts/util/ftpConfig.php
  • users class now supresses error if runtime db is not found
  • users class now saves the user db upon save
  • users class deconstructor added to save users db upon exit
  • modules support code begun (version for highly dynamic with multiple hooks etc.)
  • Changed update.php to a very basic one, as the remote capabilities are not being employed currently
  • Rudimentary module /etc/seedbox/module/basic/addUser.php for ultra simplistic user addition module (ie. custom configs, limitations).
  • Basic suspend & unsuspend capabilities (untested ALPHA)
  • skel permissions fixed

updating

bash soft.sh; php /scripts/update.php


27/03/2011

This is primarily a bug fix and code quality release.

  • Set shell to /bin/bash upon user creation
  • Set shell to /bin/bash when using userConfig
  • quotaFix.php now turns quota off before attempting quota check
  • quotaFix.php now displays quota info before and after check
  • Lots of comments added to all code files touched
  • userConfig.php $filesLimitPerGb was undefined - hardcoded 500
  • update.php minor changes
  • default version now set to 1, in preparation to take it into production
  • startRtorrent: codestyle fixes, Made error messages more clear, named rTorrent screen session to "rtorrent"