Difference between revisions of "PM Software Stack"
Line 25: | Line 25: | ||
==== 13/5/2013 ==== | ==== 13/5/2013 ==== | ||
+ | * install.sh: looks like the OVH Debian 6 wants to install apache2 with php5-geoip, so moved around the php5 modules and after their installation remove apache2 | ||
* install.sh: Added code for removing OVH installed Apache2 which is quite hard to get fully removed, and causes PMSS installation to fail | * install.sh: Added code for removing OVH installed Apache2 which is quite hard to get fully removed, and causes PMSS installation to fail | ||
Revision as of 10:00, 13 May 2013
PMSS, short of PM Software Stack is a compilation of scripts to manage single server side seedbox configuration, turning instantly and conveniently an regular Debian 5.0/6.0 server into a seedbox. The PMSS does not contain multi-server management utilities nor clustering support out-of-box, but does include PM Master GUI automatic remote update features. Debian 6.0 squeeze support is considered stable, as well as Debian 5.0. PMSS currently only supports rTorrent+ruTorrent stack. Multi-Server management is done via API with central server, with strict authentication and limited scope for security.
Contents
- 1 General features
- 2 Using PMSS
- 3 Changelog
- 3.1 Changes 2013
- 3.2 Changes 2012
- 3.2.1 7/12/2012
- 3.2.2 1/11/2012
- 3.2.3 31/10/2012
- 3.2.4 30/10/2012
- 3.2.5 14/10/2012
- 3.2.6 22/09/2012
- 3.2.7 17/09/2012
- 3.2.8 10/06/2012
- 3.2.9 22/05/2012
- 3.2.10 24/04/2012
- 3.2.11 06/04/2012
- 3.2.12 27/03/2012
- 3.2.13 21/03/2012
- 3.2.14 20/03/2012
- 3.2.15 04/03/2012
- 3.2.16 12/02/2012
- 3.2.17 10/02/2012
- 3.2.18 09/02/2012
- 3.2.19 08/02/2012
- 3.2.20 05/02/2012
- 3.2.21 04/02/2012
- 3.2.22 31/01/2012
- 3.2.23 28/01/2012
- 3.3 Changes 2011
General features
- Semi-automatic installation
- Fast and easy creation of new user accounts with quota and rTorrent resource limits, etc.
- Easy user resource reconfiguration
- Autonomous, idempotent Lighttpd configuration
- Autonomous configuration of basic server features
- Compiling rTorrent + libTorrent + Quota + XMLRPC from source.
- Multiple redundancy levels for rTorrent
- Lighttpd redundancy / autorestart
- Basic server optimizations to maximize performance
- Terminate, Suspend/Unsuspend users
- Webdav for users
- Other basic management scripts
Using PMSS
See Category:PM Software Stack Guides
Changelog
Changes 2013
13/5/2013
- install.sh: looks like the OVH Debian 6 wants to install apache2 with php5-geoip, so moved around the php5 modules and after their installation remove apache2
- install.sh: Added code for removing OVH installed Apache2 which is quite hard to get fully removed, and causes PMSS installation to fail
11/5/2013
- update-step2: Added packages required for autodl irssi
- update-step2: Added creation of /etc/autodl.cfg
8/5/2013
- trafficStats month data traffic is now for the past month, not since beginning of calendar month.
- addUser: fixed typo (use of undefined constant)
- addUser: fixed argv optionals checking
- userConfig: added several checks for user existing
- created userTrafficLimit under utils to set the traffic limit
- addUser: fixed typo in pathname for traffic limit data
- userConfig: added setting traffic limit
- userConfig: now restarts rTorrent (near instant)
- created trafficLimits cron which checks traffic limits. Communications module is still required before traffic limits can be enabled
- update-step2: added smartmontools to pkgs to be installed
20/4/2013
- trafficStats now parses 23k lines before 1st of August, 2013 because the traffic logging speed has been lowered at 17th April update -> showing just 1½ weeks initially. Time limit for that so it will not slow down always.
17/4/2013
- Added lib/traffic.php for trafficStatistics class
- Refactoring showTraffic
- stats.php shows this traffic data on info page now
- Added cron/trafficStatistics.php which saves traffic data to .trafficData periodically (once per hour)
- Decreased traffic data saving to once per 5 minutes
- Also now monitoring since beginning of the month
- showTraffic now uses the "cached" data saved by trafficStatistics
- util/userTransfer.php now puts in request for rTorrent restart after transfer is complete
- addUser optional argument for traffic limit in TBs
- update-step2 now checks, creates and chmods a bunch of required directories
13/1/2013
- Added new remote calls to serverApi, had forgotten this
- Dropped traffic logging to 4 min intervals, unfortunately this means showTraffic for 15mins is totally skewed, and maybe noticeable skew on the hourly
- updateQuotas cron now every second minute
- checkInstances now every 10mins from 15mins, making faster 2nd level redundancy for rTorrent
- changed updateQuota cron's API calls to base the call time to modify time of /scripts ie. roughly time the system was installed. This way the requests don't get pooled up at same minute from all servers
12/1/2013
Release of 10th of Jan features as well. Several bug fixes in this release.
- showTraffic now prunes erroneous data from the log
- trafficLog cron now checks for erroneous data and logs it
- updateQuotas sends the user quota and df data only once per hour
- Absolute path to iptables on setupNetwork + makeMonitoringRules so it actually works on boot time :)
10/1/2013
Waiting for release, pending server side support.
- updateQuotas now reports user quota to API server as well
- updateQuotas reports df to API server
- lib/users.php error message when issues with /home
- setupNetwork won't try to enact empty ruleset anymore
- fileCopy updated help message. This is pretty much useless nowadays in any cas and might be deprecated/removed in future release
9/1/2013
One of the DCs changed debian installation template so fixes for that. Issues with rTorrent compilation, lighttpd etc.
- Changes on install.sh package removals (apache2 not getting removed??)
- Removed munin installation from install.sh
- Added installation of pkg-config make openssl
- Fixed on update-step2 rtorrent compilation
- Install.sh now executes /scripts/update.php but does not do API key setup, it's done on the update script
5/1/2013
Quick performance related hot-fix, and one performance update.
- trafficLog now deletes the traffic log file from tmp, this was causing slowdowns etc.
- rTorrent peers better configuration, min from 14 to 6 and max from 21 to 32 per "resource block" (Resource block=250Mb of rTorrent ram). This affects only new accounts, reconfigured and recreated.
2/1/2013
Work on the new API - note this is still largely untested so far
- API create jobs dir + chmod added
- API function to make calls to server added
- API function to create authorization for remote calls added
- API make call userTraffic added
- trafficLog cron fix users check
- trafficLog reports traffic to server
- serverApi typos fixed (had not been tested before)
- serverApi create dirs bugs fixed
- serverApi logger was missing newline and has been added
- serverApi some commenting added
- fixed timestamp on scriptsInc writeLog
- trimming some elements in serverApi which due to unrelated reasons have newline at end
1/1/2013
This update also releases the changes of 7/12/2012 which were postponed due to QA reasons.
- Removal of netcat packages in update-step2 and install script
- Added count checks for users in makeMonitoringRules and recreate lighttpd rules scripts
- Copied some standard packages from update-step2 to install.sh
- Removed some apt packages from update-step2 which has been on install.sh for a *Really* long time
- Removed some irrelevant apt-get lines from install -> duplication
- Added p7zip-full package to update-step2 installation
- Fix showtraffic weekly data, was only considering ~2/5ths of the data
- ShowTraffic compares now user count, not the string returned in order to determine if there is users setup
- More uniform ShowTraffic display output, which is also programmatically easy to parse
Changes 2012
7/12/2012
This update is pending for release.
- Removal of root.cron.additional config from usage, as it does not conform to good practices
- Removal of deprecated code from update-step2 relating to vnstat and root.cron.additional
- Removing root.cron.additional on update-step2
- Removal of deprecated code from setupRootCron
- Removal of /var/run/screen/S-[username] during user termination
- Executing setupNetwork on user addition
- Added tor repos for debian squeeze
- Update-Step2 now checks squeeze repos against the actual configured, not hard coded sha1 string
- Removal of deprecated apt update from beginning of update-step2
- Some commenting and refactoring on update-step2
- Remove munin if this is a .pulsedmedia.com server
- Adduser now logs to /var/log/pmss
- Create remote api key
- Salt for local api key
- Salt for remote api key
- Enhanced algo for remote and local api key, more factors etc. making it more distinguished
- chmod api keys
- Update-Step2: if no remote api key present, remove local key and execute API key creation
- rename lib/api.php to lib/serverApi.php
- Full rewrite of serverApi, pretty much complete server api infrastructure including command map for: addUser, terminateUser, setupNetwork, recreateUser
1/11/2012
- Removed updating from OVH kernel to standard debian kernel
- No fair share / QoS rules done if OVH kernel is present
- Setup network (monitoring, QOS) on update step 2
- Changed root.cron.additional vnstat lines to conform cron.d format
- If debian 6 removal of vnstat -u cronjobs
- setupRootCron both cron force-reload and restart to ensure new cron will be enabled
- Temporarily disable all QoS & Fair share, unintentional effects
31/10/2012
- Display traffic used per user on week, daily, hourly and 15min basis. Note: 15min is slightly inaccurate due to datalogging delays (2min interval), hourly is within acceptable error margin
- Fine-tuning of fair share
- QoS is always setup and cleared.
- recreateUser: Added a check if backup dir already exists.
30/10/2012
- Fixed var/www/index.php redirection
- Added script to create traffic monitoring iptables rules
- Created /scripts/cron
- moved updateQuotas, checkInstances, checkHttpd to cron dir
- created /var/log/pmss
- Changed checkInstances + checkHttpd logging to /var/log/pmss dir
- Refactored cron tasks to utilize /etc/cron.d NOTE: If you have customized root crontab this may overwrite the changes!
- Removed "tester" from user filtration list and added srvapi, pmcseed, pmcdn to the filtration list
- Beginnings of network management cron
- If an OVH kernel is detected on Deb Squeeze, install standard and set it as default. Note: No Deb5 support yet - only changed IF vanilla kernel is not installed
- Added packages ethtool, elinks, links, lynx
- Added Basic QoS to ensure quality of service and fairness in BW sharing - enabled at boot time. Note: Does not work on OVH kernel - OVH has removed support for these in their kernel
- Cron jobs now also save their error output to log files
- Traffic logging script and /var/log/pmss/traffic directory for traffic log files.
14/10/2012
- pulsedBox authentication pruning to filter/fix the most common user errors
22/09/2012
- Fixed quota parsin in GUI when using LVM device /dev/mapper/ type addressing
- Updated installer to install rTorrent 0.9.2 and libTorrent 0.13.2
- Fixed updater setup of squeeze repos
- ruTorrent update, does include saving user preferences etc. to the new one
- Fixed a bug relating to .pulsedmedia mgmt features, incorrect order for permissions setting
- Update-step2 mild refactoring
- Removed useless apt-get install from update.php
- Added ffmpeg along with screenshots plugin
- Removed unpack plugin (way too high resource consumption)
- getfile plugin included now
- Applied quota parsing fixed to hddquota plugin and made slightly more dynamic
- Lowered time before displaying welcome page quota information
17/09/2012
- Removed proxy from soft.sh as this hasn't been included for sometime now.
- Updated rTorrent to 0.9.2 and libTorrent to 0.13.2 official releases
10/06/2012
- PEX + DHT enabled by default. DHT port same as SCGI port for now. Template changed - not the actual code, so in all versions the default has changed. Server specific configuration can disable.
22/05/2012
- Quota display bugs fixed: Servers with UUID assignment instead of device and in some cases collecting the total quota limit had last number cut off
24/04/2012
- Munin configuration updates for .pulsedmedia.com
- Updated method of checking ruTorrent version to avoid "updates" when none is needed
- More conditions for user space update, if removed user but something remains as backup / as a failure of user termination script
- Refactored a bit update-step2 fixing when .rtorrent.rc is updated
06/04/2012
- Compile rTorrent always when version mismatch during update, even if empty/not installed
- Added download speed limit when downloading via ajaxManager
- Changed chat tab to use username instead of random
27/03/2012
- Update debian 5 repos to the archive ones
- On install also enable fastcgi-php (Squeeze)
- Get quota from apt repo, and do not compile on install (Lenny support is now limited)
- Updated squeeze repos
- Slightly refactored update-step2 and install
21/03/2012
- Removed autotools + cpuload plugin from skel rutorrent
20/03/2012
- rTorrent + libtorrent version to be changed to GIT 2011/11/08 which is slightly updated from release 0.8.9, few bug fixes etc. GIT information retained if you want to check validity.
- Removal of cpuload + autotools rutorrent plugins. CPULoad being misleading at best, autotools haven't worked for ages and developer is MIA.
- Changed fetching libmediainfo packages from pulsedmedia.com instead of sourceforge.
- Choosing libmedia info + libs based on debian version. Now Debian 6.0 should be fully supported in updates.
- Creating symlink /usr/local/bin/mktorrent -> /usr/bin/mktorrent thus fixing ruTorrent torrent creation in some Debian 6.0 systems (with updated ruTorrent+rTorrent)
- Cleaning installed libtorrent
04/03/2012
- Allow key based authentication on SSH
- Pulsed Media specific management code - not added unless server is for .pulsedmedia.com
- Added srvmgmt to filtered users --- NOTICE: srvadmin user account will be deprecated in a future release
12/02/2012
- added scripts inc for files to be updated/copied on update
10/02/2012
- Fixed torrent creation from ruTorrent
- Fixed quota plugin (JSON parser has been changed to strict parsing)
- Updated quota plugin version information
- Fixed ajax manager recycle bin
- Made ruTorrent to update browser cache
- Created framework for easy user space files updating during server update (files dependant upon server upgrade)
- Removed recreateLighttpdConfig + lighttpd restart from update step 2 (should by now be in main update in 99.9%+ servers)
- Changed update.php to do lighttpd force-reload instead of restart
- Refactored update-step2 slightly: Moved specific files, and user specific stuff into single loop
- Added several packages to be installed during server installation, which were only on update. They still remain on update as well before better solution can be done (dselect)
- Filtering backup user dirs out from user list
- During scripts update, also update system packages
09/02/2012
- Reversing yesterday's update due to security concerns
- Removed installer speedups due to stability concerns
08/02/2012
- Fixed typo in install script preventing quota from being compiled
05/02/2012
- Lighttpd session "stability" upgrade, fixing issue causing session timeouts/re-requiring authentication
04/02/2012
- Added mediainfo
- Added server root document to redirect to pulsedmedia.com
- Added php5-geoip extension to be installed
- Simplified update-step2 scripts being installed :)
- Removed apt dist-upgrade from install
- Simplified (less) apt-get lines during install
- Reordered apt-get commands during install for swifter installation
- Removed some "junk" code from install
- Updated copyright text in install.sh
- Added some information on header of install.sh
- Install.sh some additional comments
- Made installation slightly quicker by usage of background processes (BETA, if your disk is extremely slow this may cause installation to fail)
- Reordering other items in install.sh to make installation faster
- Some preliminary work for supporting whitelabeling
- Removed umask, hash interval, hash max tries rtorrent configures (deprecated)
- Updated rTorrent + ruTorrent to versions 0.9.0/0.13.0
Note: Please report any issues due to the upgrade of rTorrent immediately to Pulsed Media. The patch has been tested on a small group of servers, but all situations are hard to foresee.
31/01/2012
- Fixed some typos in update-step2
28/01/2012
- Added iotop and cfv to be installed from apt repo.
- Added -m toggle for htpasswd because for some reason this is not default in Debian Lenny
- Added SSL support (self signed cert), just change http to https
- Added Webdav support, accessible from "/webdav-USERNAME/" *** BETA: Permission issues, your mileage will vary
- Removed SSL key creation from install
- Update-step2 creates SSL key automatically utilizing the server hostname, removes old deprecated key
Changes 2011
15/12/2011
- Added .irssi/config to user dirs, autojoining to #PulsedMedia on Freenode when irssi is launched
- Updated GUI IRC links, Chat tab and Adobe download links, with other small additional information and updates.
- Removed Network Storage tab from GUI because Dropbox is blocking pageviews coming from our system for unknown reason.
- Added Proxy information on GUI
- Update-step2 sees if .irssi/config exists and if it does not copies it there
30/11/2011
- Chosen rTorrent SCGI port range increased from 4k to 8k.
- "fractional blocks" for config. Resource calcs are based on number of blocks, and fractions were not allowed prior.
29/11/2011
- During update also update user crontabs. Added to both update.php and step2, to be removed from step2 after this update has seen significant adoption.
- Restarting lighttpd during update, also done doubly for now
- Moved setup api key to regular update script. In several months it will potentially be removed from update all together as it's a installation step, and included to update older nodes.
- Scripts permissions update
- updateQuotas.php -> cron to update all user quotas, as discreet user from central cron to ensure it gets updated.
- Updated root & user crontabs to reflect updateQuotas
28/11/2011
- quotaFix now removes aquota*new files to avoid collision and thus failed update via quotacheck
- userTorrents.php : Lists the count of torrents each user has, for quicker analysis usage stats
23/11/2011
- Permissions setup script added, which is utilized by recreateUser.
- recreateUser.php fixes
22/11/2011
- Created recreateUser.php which recreates home dir, preserving data+session if existing. ALPHA
19/11/2011
- Fixed saving API Key.
17/11/2011
- Suspend and Unsuspend scripts fixed
- recreateLighttpd backup fixed
- Added access format of: servername/user-username/, for example: localhost/user-johndoe/. Thus supporting non FQDN setups
- addUser.php calls recreateLighttpd instead of just adding the hostname, this ensures lighttpd config is always up-to-date.
- Added to both update steps recreateLighttpd. It will remain on step2 for the next ~6months to ensure every system out there will have the update update.php
- Removed the default "testing" torrents
- Upload slots global, by default 6 times the non-global upload slots. It is "forcefully" inserted even if template does not it have for the moment. this forcefull insert will be removed in the future.
- checkInstances now also checks rtorrent configs, and if changed lists them on /root/changedConfigs for admin review
25/8/2011
Streamlined installer for faster setups :) First release of API, just one command now. Some things moved to BETA status and considered feature complete. Many bug fixes, additional commenting etc. also added
- fixed userTransfer chowning lacking -R, thus UID was transferred for many files from the other server causing incorrect permissions
- wrapped vnstat install into if !file_exists to see if vnstat is installed already.
- added username 'srvadmin' to filter list on systemUsers
- removed from installer editing munin.conf -> if used can be done manually. Interim solution.
- put install vim as first thing on installer
- setting node hostname straight after installing vim
- moved setting up lighttpd as last thing
- scripts, skel, etc are installed earlier in installation
- SSL cert + quota as last items on setup
- Executing update-step2 on installer (so you don't need to execute updater after setup to get everything in)
- /scripts/util/setupRootCron.php -> creating root conf moved out of update.php
- setupSkelPermissions.php, same reason as root cron setup
- added setupRootCron and setupSkelPermissions to installer
- Idempotent lighttpd configurator feature completed. Backups the config first.
- Terminate user moved to beta status, as using lighttpd configurator now and restarting lighttpd.
- Added for terminate second argument, if 2nd argument is --confirm, straight accepts termination without asking for approval, for making automation easier.
- checkHttpd timer increased to 3 seconds before restarting lighttpd
- terminateUser timer increased to 5 seconds after trying to kill all user processes
- setupApiKey -> creates node key for API. Invoked upon setup.
- added authorization key
- removed /var/www/index.lighttpd.html
- added /var/www/index.php redir to pulsedmedia.com
- added beginnings of remote API for easier remote control management, by default in debug mode for now
8/6/2011
Bug fixes.
terminateUser.php:
- Was not actually completing shell userdeletion
- Added killing user processes (forgotten)
- added sleeping 2 seconds to allow time for rTorrent/other processes by user to die
- Added 2nd attempt to userdel after data removal
- Fixed some typos in texts
- Added check for user actually exists
NOTE: Still Alpha, non feature complete and not recommended for production use.
- checkHttpd.php added sleep between lighttpd kill and restart, to avoid PID file mismatch scenario
update-step2.php:
- First time being used
- update.php moved cron & ftp config stuff AFTER step2
- Install vnstat + vim (vim not by default on Debian 6)
- Setup interface for vnstat
- Configure vnstat for Debian 6
- Make vnstat cron jobs for Debian 5: twice a minute update (thus no support for 1GigE interface)
- Debian 5 add cron jobs to root.cron.additional config, create if needed
31/5/2011
- GUI welcome page checking if proper quota information was passed has been added.
- GUI welcome page malformed serialized data fix
- GUI welcome page quota percentage missing % char
- GUI welcome page quota text styles: White and location
30/5/2011
- GUI update script quota checking added (doesn't try to update when user is over quota burst limit, no more blanking GUI when update happens!)
- Users default quota information update rate has been increased to 10mins
- HDD Quota plugin commenting added
- HDD Quota plugin parser bug with bursted amounts fixed
- HDD Quota now shows USED/QUOTA instead of FREE/QUOTA. Makes more sense.
- HDD Quota "title" fixed (showing quota info) to be whole area of the meter
- Welcome page quota meter added :) Basic graph, larger and constantly visible text for quota info :)
- Basic idempotent lighttpd configurator (alpha, not suitable for production, TESTING only)
23/5/2011
- terminateUser.php http password deletion fix, text fixes, executability.
- checkHttpd.php output lighttpd output, killall -9 lighttpd to ensure shutdown, http fetch mute removed so error gets logged
- update.php: removed apt updates because soft.sh already does this
- crons: increased check http to 4 minute intervals (avoid http timeout period issues)
18/5/2011
Temporary fix to bypass userDb until fully utilized. You may use this update or use /scripts/util/populateUserDb.php to fix the issue.
- Changed listUsers.php to always use systemUsers.php, as fixing userDb saving introduced a bug where ie. checkInstances was not working fully (using userDb, which lacks most users in most servers)
16/5/2011
Few bug fixes and minor features introduced.
- Update script order of items.
- Update script apt-get options for additional package installation lacked auto confirm.
- Update script support for "step 2", /scripts/util/update-step2.php is included if present, thus supporting dynamic updates
- Update script execute binary line added, so it can actually be executed simply by ./update.php
- /scripts permission fix, only root should have anykind of access (no execute or read for others)
- Support for root additional items via /etc/seedbox/config/root.cron.additional
- Add user script minor cleanup, additional commenting.
- Suspend script comments
- simple terminate user script, ALPHA -> not feature complete yet.
Updating from 27/03/2011 release or earlier
cd ~; bash soft.sh; php /scripts/update.php
Updating from 15/5/2011 release
php /scripts/update.php
15/5/2011
Couple important new features introduced, several bug fixes.
- Added lighttpd status checking and auto-restart (very rudimentary)
- Fixed typo in /scripts/util/userConfig.php
- Added /etc/seedbox/config/root.cron
- /scripts/util/quotaFix.php now echoes results (also turns quota off, no need to turn it off first)
- proftpd config update (remote) with MaxClientsPerUser, MaxClientsPerHost, run /scripts/util/ftpConfig.php
- users class now supresses error if runtime db is not found
- users class now saves the user db upon save
- users class deconstructor added to save users db upon exit
- modules support code begun (version for highly dynamic with multiple hooks etc.)
- Changed update.php to a very basic one, as the remote capabilities are not being employed currently
- Rudimentary module /etc/seedbox/module/basic/addUser.php for ultra simplistic user addition module (ie. custom configs, limitations).
- Basic suspend & unsuspend capabilities (untested ALPHA)
- skel permissions fixed
updating
bash soft.sh; php /scripts/update.php
27/03/2011
This is primarily a bug fix and code quality release.
- Set shell to /bin/bash upon user creation
- Set shell to /bin/bash when using userConfig
- quotaFix.php now turns quota off before attempting quota check
- quotaFix.php now displays quota info before and after check
- Lots of comments added to all code files touched
- userConfig.php $filesLimitPerGb was undefined - hardcoded 500
- update.php minor changes
- default version now set to 1, in preparation to take it into production
- startRtorrent: codestyle fixes, Made error messages more clear, named rTorrent screen session to "rtorrent"