PM Software Stack

From Pulsed Media Wiki
Revision as of 12:09, 21 December 2013 by Nucode (talk | contribs)

PMSS, short of PM Software Stack is a compilation of scripts to manage single server side seedbox configuration, turning instantly and conveniently an regular Debian 5.0/6.0 server into a seedbox. The PMSS does not contain multi-server management utilities nor clustering support out-of-box, but does include PM Master GUI automatic remote update features. Debian 6.0 squeeze support is considered stable, as well as Debian 5.0. PMSS currently only supports rTorrent+ruTorrent stack. Multi-Server management is done via API with central server, with strict authentication and limited scope for security.

General features


Using PMSS

See Category:PM Software Stack Guides

Changelog

Changes 2013

21/12/2013

  • userTransfer.php: Tries to retry 20 times with 15-120sec delay. Necessary since OVH has introduced new bw limitations: Connections are cut at random.

03/10/2013

  • install.sh: updated libxmlrpc URI to the new one.

29/9/2013

  • update-step2: Added wheezy repos
  • update-step2: Made wheezy repo update
  • update-step2: Some refactoring in regards of repo updates
  • cron/trafficLog: Updated traffic parsing to support wheezy
  • install.sh: Moved apt-get install rar unrar to it's own command due to OVH wheezy template issues
  • install.sh: Removed some commented out things, blast from past type of things
  • install.sh: Updated queue read ahead to double size
  • install.sh: Updated header comments

18/5/2013

  • install.sh: moving around php5 installation due to continuing OVH template issues

13/5/2013

  • install.sh: looks like the OVH Debian 6 wants to install apache2 with php5-geoip, so moved around the php5 modules and after their installation remove apache2
  • install.sh: Added code for removing OVH installed Apache2 which is quite hard to get fully removed, and causes PMSS installation to fail

11/5/2013

  • update-step2: Added packages required for autodl irssi
  • update-step2: Added creation of /etc/autodl.cfg

8/5/2013

  • trafficStats month data traffic is now for the past month, not since beginning of calendar month.
  • addUser: fixed typo (use of undefined constant)
  • addUser: fixed argv optionals checking
  • userConfig: added several checks for user existing
  • created userTrafficLimit under utils to set the traffic limit
  • addUser: fixed typo in pathname for traffic limit data
  • userConfig: added setting traffic limit
  • userConfig: now restarts rTorrent (near instant)
  • created trafficLimits cron which checks traffic limits. Communications module is still required before traffic limits can be enabled
  • update-step2: added smartmontools to pkgs to be installed

20/4/2013

  • trafficStats now parses 23k lines before 1st of August, 2013 because the traffic logging speed has been lowered at 17th April update -> showing just 1½ weeks initially. Time limit for that so it will not slow down always.

17/4/2013

  • Added lib/traffic.php for trafficStatistics class
  • Refactoring showTraffic
  • stats.php shows this traffic data on info page now
  • Added cron/trafficStatistics.php which saves traffic data to .trafficData periodically (once per hour)
  • Decreased traffic data saving to once per 5 minutes
  • Also now monitoring since beginning of the month
  • showTraffic now uses the "cached" data saved by trafficStatistics
  • util/userTransfer.php now puts in request for rTorrent restart after transfer is complete
  • addUser optional argument for traffic limit in TBs
  • update-step2 now checks, creates and chmods a bunch of required directories

13/1/2013

  • Added new remote calls to serverApi, had forgotten this
  • Dropped traffic logging to 4 min intervals, unfortunately this means showTraffic for 15mins is totally skewed, and maybe noticeable skew on the hourly
  • updateQuotas cron now every second minute
  • checkInstances now every 10mins from 15mins, making faster 2nd level redundancy for rTorrent
  • changed updateQuota cron's API calls to base the call time to modify time of /scripts ie. roughly time the system was installed. This way the requests don't get pooled up at same minute from all servers

12/1/2013

Release of 10th of Jan features as well. Several bug fixes in this release.

  • showTraffic now prunes erroneous data from the log
  • trafficLog cron now checks for erroneous data and logs it
  • updateQuotas sends the user quota and df data only once per hour
  • Absolute path to iptables on setupNetwork + makeMonitoringRules so it actually works on boot time :)

10/1/2013

Waiting for release, pending server side support.

  • updateQuotas now reports user quota to API server as well
  • updateQuotas reports df to API server
  • lib/users.php error message when issues with /home
  • setupNetwork won't try to enact empty ruleset anymore
  • fileCopy updated help message. This is pretty much useless nowadays in any cas and might be deprecated/removed in future release

9/1/2013

One of the DCs changed debian installation template so fixes for that. Issues with rTorrent compilation, lighttpd etc.

  • Changes on install.sh package removals (apache2 not getting removed??)
  • Removed munin installation from install.sh
  • Added installation of pkg-config make openssl
  • Fixed on update-step2 rtorrent compilation
  • Install.sh now executes /scripts/update.php but does not do API key setup, it's done on the update script

5/1/2013

Quick performance related hot-fix, and one performance update.

  • trafficLog now deletes the traffic log file from tmp, this was causing slowdowns etc.
  • rTorrent peers better configuration, min from 14 to 6 and max from 21 to 32 per "resource block" (Resource block=250Mb of rTorrent ram). This affects only new accounts, reconfigured and recreated.


2/1/2013

Work on the new API - note this is still largely untested so far

  • API create jobs dir + chmod added
  • API function to make calls to server added
  • API function to create authorization for remote calls added
  • API make call userTraffic added
  • trafficLog cron fix users check
  • trafficLog reports traffic to server
  • serverApi typos fixed (had not been tested before)
  • serverApi create dirs bugs fixed
  • serverApi logger was missing newline and has been added
  • serverApi some commenting added
  • fixed timestamp on scriptsInc writeLog
  • trimming some elements in serverApi which due to unrelated reasons have newline at end

1/1/2013

This update also releases the changes of 7/12/2012 which were postponed due to QA reasons.

  • Removal of netcat packages in update-step2 and install script
  • Added count checks for users in makeMonitoringRules and recreate lighttpd rules scripts
  • Copied some standard packages from update-step2 to install.sh
  • Removed some apt packages from update-step2 which has been on install.sh for a *Really* long time
  • Removed some irrelevant apt-get lines from install -> duplication
  • Added p7zip-full package to update-step2 installation
  • Fix showtraffic weekly data, was only considering ~2/5ths of the data
  • ShowTraffic compares now user count, not the string returned in order to determine if there is users setup
  • More uniform ShowTraffic display output, which is also programmatically easy to parse

Changes 2012

7/12/2012

This update is pending for release.

  • Removal of root.cron.additional config from usage, as it does not conform to good practices
  • Removal of deprecated code from update-step2 relating to vnstat and root.cron.additional
  • Removing root.cron.additional on update-step2
  • Removal of deprecated code from setupRootCron
  • Removal of /var/run/screen/S-[username] during user termination
  • Executing setupNetwork on user addition
  • Added tor repos for debian squeeze
  • Update-Step2 now checks squeeze repos against the actual configured, not hard coded sha1 string
  • Removal of deprecated apt update from beginning of update-step2
  • Some commenting and refactoring on update-step2
  • Remove munin if this is a .pulsedmedia.com server
  • Adduser now logs to /var/log/pmss
  • Create remote api key
  • Salt for local api key
  • Salt for remote api key
  • Enhanced algo for remote and local api key, more factors etc. making it more distinguished
  • chmod api keys
  • Update-Step2: if no remote api key present, remove local key and execute API key creation
  • rename lib/api.php to lib/serverApi.php
  • Full rewrite of serverApi, pretty much complete server api infrastructure including command map for: addUser, terminateUser, setupNetwork, recreateUser

1/11/2012

  • Removed updating from OVH kernel to standard debian kernel
  • No fair share / QoS rules done if OVH kernel is present
  • Setup network (monitoring, QOS) on update step 2
  • Changed root.cron.additional vnstat lines to conform cron.d format
  • If debian 6 removal of vnstat -u cronjobs
  • setupRootCron both cron force-reload and restart to ensure new cron will be enabled
  • Temporarily disable all QoS & Fair share, unintentional effects

31/10/2012

  • Display traffic used per user on week, daily, hourly and 15min basis. Note: 15min is slightly inaccurate due to datalogging delays (2min interval), hourly is within acceptable error margin
  • Fine-tuning of fair share
  • QoS is always setup and cleared.
  • recreateUser: Added a check if backup dir already exists.

30/10/2012

  • Fixed var/www/index.php redirection
  • Added script to create traffic monitoring iptables rules
  • Created /scripts/cron
  • moved updateQuotas, checkInstances, checkHttpd to cron dir
  • created /var/log/pmss
  • Changed checkInstances + checkHttpd logging to /var/log/pmss dir
  • Refactored cron tasks to utilize /etc/cron.d NOTE: If you have customized root crontab this may overwrite the changes!
  • Removed "tester" from user filtration list and added srvapi, pmcseed, pmcdn to the filtration list
  • Beginnings of network management cron
  • If an OVH kernel is detected on Deb Squeeze, install standard and set it as default. Note: No Deb5 support yet - only changed IF vanilla kernel is not installed
  • Added packages ethtool, elinks, links, lynx
  • Added Basic QoS to ensure quality of service and fairness in BW sharing - enabled at boot time. Note: Does not work on OVH kernel - OVH has removed support for these in their kernel
  • Cron jobs now also save their error output to log files
  • Traffic logging script and /var/log/pmss/traffic directory for traffic log files.


14/10/2012

  • pulsedBox authentication pruning to filter/fix the most common user errors

22/09/2012

  • Fixed quota parsin in GUI when using LVM device /dev/mapper/ type addressing
  • Updated installer to install rTorrent 0.9.2 and libTorrent 0.13.2
  • Fixed updater setup of squeeze repos
  • ruTorrent update, does include saving user preferences etc. to the new one
  • Fixed a bug relating to .pulsedmedia mgmt features, incorrect order for permissions setting
  • Update-step2 mild refactoring
  • Removed useless apt-get install from update.php
  • Added ffmpeg along with screenshots plugin
  • Removed unpack plugin (way too high resource consumption)
  • getfile plugin included now
  • Applied quota parsing fixed to hddquota plugin and made slightly more dynamic
  • Lowered time before displaying welcome page quota information

17/09/2012

  • Removed proxy from soft.sh as this hasn't been included for sometime now.
  • Updated rTorrent to 0.9.2 and libTorrent to 0.13.2 official releases

10/06/2012

  • PEX + DHT enabled by default. DHT port same as SCGI port for now. Template changed - not the actual code, so in all versions the default has changed. Server specific configuration can disable.

22/05/2012

  • Quota display bugs fixed: Servers with UUID assignment instead of device and in some cases collecting the total quota limit had last number cut off

24/04/2012

  • Munin configuration updates for .pulsedmedia.com
  • Updated method of checking ruTorrent version to avoid "updates" when none is needed
  • More conditions for user space update, if removed user but something remains as backup / as a failure of user termination script
  • Refactored a bit update-step2 fixing when .rtorrent.rc is updated

06/04/2012

  • Compile rTorrent always when version mismatch during update, even if empty/not installed
  • Added download speed limit when downloading via ajaxManager
  • Changed chat tab to use username instead of random

27/03/2012

  • Update debian 5 repos to the archive ones
  • On install also enable fastcgi-php (Squeeze)
  • Get quota from apt repo, and do not compile on install (Lenny support is now limited)
  • Updated squeeze repos
  • Slightly refactored update-step2 and install

21/03/2012

  • Removed autotools + cpuload plugin from skel rutorrent

20/03/2012

  • rTorrent + libtorrent version to be changed to GIT 2011/11/08 which is slightly updated from release 0.8.9, few bug fixes etc. GIT information retained if you want to check validity.
  • Removal of cpuload + autotools rutorrent plugins. CPULoad being misleading at best, autotools haven't worked for ages and developer is MIA.
  • Changed fetching libmediainfo packages from pulsedmedia.com instead of sourceforge.
  • Choosing libmedia info + libs based on debian version. Now Debian 6.0 should be fully supported in updates.
  • Creating symlink /usr/local/bin/mktorrent -> /usr/bin/mktorrent thus fixing ruTorrent torrent creation in some Debian 6.0 systems (with updated ruTorrent+rTorrent)
  • Cleaning installed libtorrent

04/03/2012

  • Allow key based authentication on SSH
  • Pulsed Media specific management code - not added unless server is for .pulsedmedia.com
  • Added srvmgmt to filtered users --- NOTICE: srvadmin user account will be deprecated in a future release

12/02/2012

  • added scripts inc for files to be updated/copied on update

10/02/2012

  • Fixed torrent creation from ruTorrent
  • Fixed quota plugin (JSON parser has been changed to strict parsing)
  • Updated quota plugin version information
  • Fixed ajax manager recycle bin
  • Made ruTorrent to update browser cache
  • Created framework for easy user space files updating during server update (files dependant upon server upgrade)
  • Removed recreateLighttpdConfig + lighttpd restart from update step 2 (should by now be in main update in 99.9%+ servers)
  • Changed update.php to do lighttpd force-reload instead of restart
  • Refactored update-step2 slightly: Moved specific files, and user specific stuff into single loop
  • Added several packages to be installed during server installation, which were only on update. They still remain on update as well before better solution can be done (dselect)
  • Filtering backup user dirs out from user list
  • During scripts update, also update system packages

09/02/2012

  • Reversing yesterday's update due to security concerns
  • Removed installer speedups due to stability concerns

08/02/2012

  • Fixed typo in install script preventing quota from being compiled

05/02/2012

  • Lighttpd session "stability" upgrade, fixing issue causing session timeouts/re-requiring authentication

04/02/2012

  • Added mediainfo
  • Added server root document to redirect to pulsedmedia.com
  • Added php5-geoip extension to be installed
  • Simplified update-step2 scripts being installed :)
  • Removed apt dist-upgrade from install
  • Simplified (less) apt-get lines during install
  • Reordered apt-get commands during install for swifter installation
  • Removed some "junk" code from install
  • Updated copyright text in install.sh
  • Added some information on header of install.sh
  • Install.sh some additional comments
  • Made installation slightly quicker by usage of background processes (BETA, if your disk is extremely slow this may cause installation to fail)
  • Reordering other items in install.sh to make installation faster
  • Some preliminary work for supporting whitelabeling
  • Removed umask, hash interval, hash max tries rtorrent configures (deprecated)
  • Updated rTorrent + ruTorrent to versions 0.9.0/0.13.0

Note: Please report any issues due to the upgrade of rTorrent immediately to Pulsed Media. The patch has been tested on a small group of servers, but all situations are hard to foresee.

31/01/2012

  • Fixed some typos in update-step2

28/01/2012

  • Added iotop and cfv to be installed from apt repo.
  • Added -m toggle for htpasswd because for some reason this is not default in Debian Lenny
  • Added SSL support (self signed cert), just change http to https
  • Added Webdav support, accessible from "/webdav-USERNAME/" *** BETA: Permission issues, your mileage will vary
  • Removed SSL key creation from install
  • Update-step2 creates SSL key automatically utilizing the server hostname, removes old deprecated key

Changes 2011

15/12/2011

  • Added .irssi/config to user dirs, autojoining to #PulsedMedia on Freenode when irssi is launched
  • Updated GUI IRC links, Chat tab and Adobe download links, with other small additional information and updates.
  • Removed Network Storage tab from GUI because Dropbox is blocking pageviews coming from our system for unknown reason.
  • Added Proxy information on GUI
  • Update-step2 sees if .irssi/config exists and if it does not copies it there

30/11/2011

  • Chosen rTorrent SCGI port range increased from 4k to 8k.
  • "fractional blocks" for config. Resource calcs are based on number of blocks, and fractions were not allowed prior.

29/11/2011

  • During update also update user crontabs. Added to both update.php and step2, to be removed from step2 after this update has seen significant adoption.
  • Restarting lighttpd during update, also done doubly for now
  • Moved setup api key to regular update script. In several months it will potentially be removed from update all together as it's a installation step, and included to update older nodes.
  • Scripts permissions update
  • updateQuotas.php -> cron to update all user quotas, as discreet user from central cron to ensure it gets updated.
  • Updated root & user crontabs to reflect updateQuotas

28/11/2011

  • quotaFix now removes aquota*new files to avoid collision and thus failed update via quotacheck
  • userTorrents.php : Lists the count of torrents each user has, for quicker analysis usage stats

23/11/2011

  • Permissions setup script added, which is utilized by recreateUser.
  • recreateUser.php fixes

22/11/2011

  • Created recreateUser.php which recreates home dir, preserving data+session if existing. ALPHA

19/11/2011

  • Fixed saving API Key.

17/11/2011

  • Suspend and Unsuspend scripts fixed
  • recreateLighttpd backup fixed
  • Added access format of: servername/user-username/, for example: localhost/user-johndoe/. Thus supporting non FQDN setups
  • addUser.php calls recreateLighttpd instead of just adding the hostname, this ensures lighttpd config is always up-to-date.
  • Added to both update steps recreateLighttpd. It will remain on step2 for the next ~6months to ensure every system out there will have the update update.php
  • Removed the default "testing" torrents
  • Upload slots global, by default 6 times the non-global upload slots. It is "forcefully" inserted even if template does not it have for the moment. this forcefull insert will be removed in the future.
  • checkInstances now also checks rtorrent configs, and if changed lists them on /root/changedConfigs for admin review

25/8/2011

Streamlined installer for faster setups :) First release of API, just one command now. Some things moved to BETA status and considered feature complete. Many bug fixes, additional commenting etc. also added

  • fixed userTransfer chowning lacking -R, thus UID was transferred for many files from the other server causing incorrect permissions
  • wrapped vnstat install into if !file_exists to see if vnstat is installed already.
  • added username 'srvadmin' to filter list on systemUsers
  • removed from installer editing munin.conf -> if used can be done manually. Interim solution.
  • put install vim as first thing on installer
  • setting node hostname straight after installing vim
  • moved setting up lighttpd as last thing
  • scripts, skel, etc are installed earlier in installation
  • SSL cert + quota as last items on setup
  • Executing update-step2 on installer (so you don't need to execute updater after setup to get everything in)
  • /scripts/util/setupRootCron.php -> creating root conf moved out of update.php
  • setupSkelPermissions.php, same reason as root cron setup
  • added setupRootCron and setupSkelPermissions to installer
  • Idempotent lighttpd configurator feature completed. Backups the config first.
  • Terminate user moved to beta status, as using lighttpd configurator now and restarting lighttpd.
  • Added for terminate second argument, if 2nd argument is --confirm, straight accepts termination without asking for approval, for making automation easier.
  • checkHttpd timer increased to 3 seconds before restarting lighttpd
  • terminateUser timer increased to 5 seconds after trying to kill all user processes
  • setupApiKey -> creates node key for API. Invoked upon setup.
  • added authorization key
  • removed /var/www/index.lighttpd.html
  • added /var/www/index.php redir to pulsedmedia.com
  • added beginnings of remote API for easier remote control management, by default in debug mode for now


8/6/2011

Bug fixes.

terminateUser.php:

  • Was not actually completing shell userdeletion
  • Added killing user processes (forgotten)
  • added sleeping 2 seconds to allow time for rTorrent/other processes by user to die
  • Added 2nd attempt to userdel after data removal
  • Fixed some typos in texts
  • Added check for user actually exists

NOTE: Still Alpha, non feature complete and not recommended for production use.

  • checkHttpd.php added sleep between lighttpd kill and restart, to avoid PID file mismatch scenario

update-step2.php:

  • First time being used
  • update.php moved cron & ftp config stuff AFTER step2
  • Install vnstat + vim (vim not by default on Debian 6)
  • Setup interface for vnstat
  • Configure vnstat for Debian 6
  • Make vnstat cron jobs for Debian 5: twice a minute update (thus no support for 1GigE interface)
  • Debian 5 add cron jobs to root.cron.additional config, create if needed

31/5/2011

  • GUI welcome page checking if proper quota information was passed has been added.
  • GUI welcome page malformed serialized data fix
  • GUI welcome page quota percentage missing % char
  • GUI welcome page quota text styles: White and location

30/5/2011

  • GUI update script quota checking added (doesn't try to update when user is over quota burst limit, no more blanking GUI when update happens!)
  • Users default quota information update rate has been increased to 10mins
  • HDD Quota plugin commenting added
  • HDD Quota plugin parser bug with bursted amounts fixed
  • HDD Quota now shows USED/QUOTA instead of FREE/QUOTA. Makes more sense.
  • HDD Quota "title" fixed (showing quota info) to be whole area of the meter
  • Welcome page quota meter added :) Basic graph, larger and constantly visible text for quota info :)
  • Basic idempotent lighttpd configurator (alpha, not suitable for production, TESTING only)

23/5/2011

  • terminateUser.php http password deletion fix, text fixes, executability.
  • checkHttpd.php output lighttpd output, killall -9 lighttpd to ensure shutdown, http fetch mute removed so error gets logged
  • update.php: removed apt updates because soft.sh already does this
  • crons: increased check http to 4 minute intervals (avoid http timeout period issues)

18/5/2011

Temporary fix to bypass userDb until fully utilized. You may use this update or use /scripts/util/populateUserDb.php to fix the issue.

  • Changed listUsers.php to always use systemUsers.php, as fixing userDb saving introduced a bug where ie. checkInstances was not working fully (using userDb, which lacks most users in most servers)

16/5/2011

Few bug fixes and minor features introduced.

  • Update script order of items.
  • Update script apt-get options for additional package installation lacked auto confirm.
  • Update script support for "step 2", /scripts/util/update-step2.php is included if present, thus supporting dynamic updates
  • Update script execute binary line added, so it can actually be executed simply by ./update.php
  • /scripts permission fix, only root should have anykind of access (no execute or read for others)
  • Support for root additional items via /etc/seedbox/config/root.cron.additional
  • Add user script minor cleanup, additional commenting.
  • Suspend script comments
  • simple terminate user script, ALPHA -> not feature complete yet.

Updating from 27/03/2011 release or earlier

cd ~; bash soft.sh; php /scripts/update.php

Updating from 15/5/2011 release

php /scripts/update.php


15/5/2011

Couple important new features introduced, several bug fixes.

  • Added lighttpd status checking and auto-restart (very rudimentary)
  • Fixed typo in /scripts/util/userConfig.php
  • Added /etc/seedbox/config/root.cron
  • /scripts/util/quotaFix.php now echoes results (also turns quota off, no need to turn it off first)
  • proftpd config update (remote) with MaxClientsPerUser, MaxClientsPerHost, run /scripts/util/ftpConfig.php
  • users class now supresses error if runtime db is not found
  • users class now saves the user db upon save
  • users class deconstructor added to save users db upon exit
  • modules support code begun (version for highly dynamic with multiple hooks etc.)
  • Changed update.php to a very basic one, as the remote capabilities are not being employed currently
  • Rudimentary module /etc/seedbox/module/basic/addUser.php for ultra simplistic user addition module (ie. custom configs, limitations).
  • Basic suspend & unsuspend capabilities (untested ALPHA)
  • skel permissions fixed

updating

bash soft.sh; php /scripts/update.php


27/03/2011

This is primarily a bug fix and code quality release.

  • Set shell to /bin/bash upon user creation
  • Set shell to /bin/bash when using userConfig
  • quotaFix.php now turns quota off before attempting quota check
  • quotaFix.php now displays quota info before and after check
  • Lots of comments added to all code files touched
  • userConfig.php $filesLimitPerGb was undefined - hardcoded 500
  • update.php minor changes
  • default version now set to 1, in preparation to take it into production
  • startRtorrent: codestyle fixes, Made error messages more clear, named rTorrent screen session to "rtorrent"