Icecast -kh branches
This is the current set of branches of ices2 and icecast2. They extend on the offical
release (currently 2.3.2) with features that may be (if found to be working out well) merged
into the next offical release. You can contact me karl <at> xiph.org
Website has moved to github HERE
git version control Here
Trunk build 01/06/2010 win32
- add generic scattered IO routines, listeners wanting FLV wrapping now use this which
saves a lot of memory copying and therefore load.
- build setup for mingw32. This should help with library updates for us, drop GUI build
- relay restarting fixes included for certain error cases. relays are treated as
failed if they stop within 60 seconds and skip on to next server if specified.
- a better trigger for on-demand relay if fallback has listeners.
- on reload when changeowner used, allow for listening sockets to be reopend but
prevent closing priviledged ports unless missing from xml
- prevent unescape routine from creating non-printable chars
- ignore per-mount username when source is using shoutcast protocol.
- URL auth can now accept a "ice-username: ..." in the response headers for setting the
username. For setups that use query args for auth
- expand args applied to xsl requests, url encoded now.
- modify k and m in bitrates to be on 1000 not 1024
- accesslog modification to parsing, optionally set <type> in <accesslog> to CLF-ESC
to escape-encode strings instead of using "".
- if falling back to file, the bitrate used takes, in order of priority, incoming
bitrate, limit-rate and then [value] in filename eg /fallback-
- fix a large ogg page bug, flac is probably the only one showing this up currently
- fix a couple of rare crash cases with file serving clients.
- prevent intro file sending when override triggered.
- make access log report query args on the request, but truncate the strings if too long.
- minor change to send limiter
- move incoming rate limiter to source client specific code.
- Add server-wide redirect tags.
- use per-source stats handles for updates, reduces lookups and avoids taking a shared lock.
- minor log message changes.
- fix fallback trigger if source timeout occurs
- fix possible sources count inconsistency with failed relays
- entity expansion in stats conversion routines added
- fixup debug allocation code. not normally used.
- stats updates.
- Split stats lock into global and source stats locks, reduces contention.
- Updated internal API for using a stats handle for quick lookup when updating several stats at once.
- Make metadata_updated stat to be sent last in the initial listing, we would want that to appear after title, useful as a trigger for updates.
- make NEW also report the content type after the mountpoint
- re-work metadata insert merge with shoutcast inserts for mp3/aac, do metadata followed by stream data instead of the other way around, reduces code.
- fix url decode crash bug with %invalid code.
- prevent yp add if required data is missing.
- allow for all args to be supplied to xsl transforms from request not just mount
- xsl transforms are written direct to refbufs, so should avoid a memory copy on each transform, should help in cases of large data sets.
- Add redirect tag on <mount>
- change default to allow ogg matadata via url (for consistency with trunk).
- have master relays supply auth by default.
- allow for quick responses to be tried and possible close the connection, saves rescheduling the worker for just closing the socket.
- streamlist.txt was not getting tried if /admin/streams failed. IE kh slave to a 2.3.2 master.
- reduce duration of when the config write lock is held when doing reload.
- a number of tunings for rescheduling clients (source and stats mainly).
- small reworking of client_t creation and accounting.
- missing redirect new listener case not handled.
- source count could get out of sync in certainly failing relay cases.
- a few cases of memory leaks fixed, not commonly used but can accumulate in certain setups, eg intros via auth or reloading frequently.
- automatically set nofile and corefile size limits to max allowed, for debug
- kh29.1 is a win32 specific fix. work around a string termination bug in the compiler
- prevent crash with very small burst size (eg 0).
- fix possible race with relay restart and headers used in stats update
- fix possible mpeg issues causing a crash or getting stuck in an allocation loop on bad input.
- An FLV listener could trigger a corruption in the metadata for non-FLV listeners. More likely
to occur when starting on-demand relays.
- relays that have only been connected for a few seconds will be treated as if it had failed to
restart and listeners will be droped or moved.
- fix possible server stall if reload requested and auth listeners were pending on auth queue
- Fix a number of cases where listeners kept a relay from restarting because of an incorrect
internal state, a stuck relay.
- revert to reserving a new relay source later in client processing. The relay is still installed
immediately (with client) but the source could clash with another already reserved.
- stats cleanup on disabled or failed relay start
- on-demand relays minimum transfer set to 2Meg
- make rejected_mount and stream_auth work properly again for URL auth
- possible listener cleanup race if auth used at exit. We now keep listener on the worker even
if inactive until complete.
- expanded FLV metadata, double and bool settings, adds audiocodecid, audiosamplerate, stereo and
various entries from stats.
- slave relay could of been redirected, now it is marked as a slave by the master so bypassess
various limit checks.
- If auth server does not respond then disable URL requests for listeners for 60 seconds. Keeps
icecast from clogging up if link to auth server breaks. Some people use auth for non-auth
accounting or listener intro, so extra option added to allow those by default (option
presume_innocent set to yes to allow listener even if auth not active).
- Make auth refcount change based on threads holding it not clients, simplifies the shutdown or
busy startup case.
- move non-ogg metadata update code into one place, and fix-up use of inline URL parsing and
possible short write in rare cases.
- slow down listener thread (therefore reduce new incoming listeners) if relay starts up, more
so if many relays start or auth queues are full.
- fix ogg stream dump-file.
- new listeners (with queryargs or user:pass) redirected because of icecast will
have their supplied auth details added to new location.
- add error code to recoverable list for accept, and prevent listener thread slow down in those
cases, seems to affect FreeBSD more than others although it could affect others.
- revert listener stat names for listclients to 2.3.2 case for now
- code cleanup and minor speeups
- minor log message cleanups.
- fix possible short send shoutcast metadata inserts getting corrupted
- fix a few possible flv issues when metadata is missing.
- add filter routine for mpeg parsing, used for file inputs like intro or fallback
- add min-queue-size mount setting, defaults to burst size so backward compatible.
players can request anything up to min-queue-setting with the initial-burst
header or burst= arg
- worker shutdown forces clients to process, should allow for quicker termination
or moving to another worker.
- stats client tidy up, prevent a few more stats going through to slave, report
slave auth failure properly
- introduce SYNC flag to source to keep internal counts consistent when moving or
- add proxy/cache avoidance headers
- drop start timer from relay as we use the more general client schedule timer
- removed the relay recheck each second as the relays are now installed and started
via the clients on the workers. saves work with many relays.
- possible crash point on relay details change, or xml reload
- offset in file serving was always 0 even if range was requested.
- add in the 7.xsl hack for compatability
- A few internal code and log message cleanups.
- Fix some long standing stats bugs, possible junk data referenced and source stats deletion
which could make streams hidden if no mount definition specified
- Fix possible incorrect socket closure if dealing with relays, this can show as a busy icecast
as one of the workers is often looping too quickly because of this.
- Add mpeg TS markers to mpeg parsing, and apply mpeg check initially then fall back to
fixed size buffers.
- use the tree locks instead of stats lock, this should lessen contention with many
stats updates, typicaly of setups with many streams.
- rework previous non-ogg metadata short-write fix, was not complete.
- previous non-ogg metadata fix exposed problem if file fallback used.
- remove possible race waking up a worker, could result in crash.
- add <max-stream-duration> to <mount> (in seconds), could even be used to cycle
dump files on a relay.
- leave public setting as-is when on-demand relay stops.
- minor legacy code cleanup
- fix possible content corruption on shoutcast metadata inserted streams if a short
send occurs during metadata sending.
- do not list non-hidden streams in /admin/stream[list|streams] and make sure auth
applies to /admin/streams.
- allow for the so-sndbuf option per-mount as well as on all connections.
- old style bind-address setting (not listen-socket) was not working
- allow a negative ban-client value to use the internal deny block of IPs to prevent
more than 1 IP connecting at the same time.
- small code changes for main worker thread
- FLV clients could skip frames on a truncated write, leading to playback problems.
- various client scheduling tunings, allows quicker reschedule when only some data was written.
This seems to affect win32 more than others, especially with burst
- fix shoutcast-inline metadata bug involving a URL
- per listener mpeg checking only verifies 1 frame as queue is already checked
- set non-ogg intial blocksize to 1400 again, now that blocks can be more flexible in size
- filter out multiple / in requests
- the 2.3 log archive setting wasn't handled correctly.
- certain relay configuration (multiple addresses) could leak a file descriptor on failure.
- possible NULL pointer crash in metadata update.
- possible sources count not decreased if relay fails.
- fix for possible relay change restart
- worker lock changes, now we only take the lock when adding clients. This reduces some lock
order complexity when moving clients around and means that no locking is done for most of the time.
- relay changes are tagged and changed by the client functions.
- compile cleanups
- Allow for per-listener wrapping of existing mp3/aac streams when requested with
a type=.flv as a query arg. The actual trigger could be anything we want but this
allows for using the existing mountpoint, with easy mapping for URL auth and for
matching an extension in the player. Works with intro and fallback handling.
- mpeg ADTS parser is more complete, handles more cases, avoiding bad input cases
- fix possible bad pointer in ogg headers case.
- fix for shoutcast-style metadata writes being truncated.
- fixes for non-ogg buffer alignment causing resync or stalled streams.
- prevent a failed stream list request from dropping current relays
- handle relay startup shutdown better for memory usage and listeners
- use Host header if available for buildm3u
- if an inactive on-demand relay detects listeners on fallback then start relay
but do not move listeners until the connection is complete.
- fix race condition with restarting non-ogg relays and stats being generated
- make auth queue limit 150 before rejection.
- A number of relay related fixes. these typically occur when a relay is either
restarted or was removed (eg disappeared from a master). There is also a
possible case of long held locking that is now gone.
- add <ban-client>N</ban-client> mount option to add client IP to the
internal banned list for N seconds. subsequent attempts that have not timed out extend
the ban period. A stat count of banned entries is produced each second
- reload from admin page was not restarting listen thread if signalfd used.
- increase default queue block size on non-ogg streams to 2900
- mpeg parsing fix for odd cases. This could appear as corrupt or stalled streams
- fix listing of stats and listener details on fallback files
- fix Location header reference from auth url
- Add mp3/aac sync marker alignment code. This makes the internal buffers contain
whole frames. allows for better transistion when moving listeners.
- The number of logs is not limited to 25 now
- small changes to client scheduling
- removed format url lock, source lock is always taken, so this is not needed
- Add <skip-accesslog> boolean to per-mount settings. Prevents logging listener
client details to access log
- evaluate bandwidth usage when adding to the global bitrate, and change throttle
indicator if max bandwidth specified. listeners have varying degrees of
scheduling limits applied. lagging listeners get delayed first, then all do.
- fix long standing bug which could cause a crash on a bad xsl.
- report line numbers on xml errors.
- small work-around to handle odd handling of ICY response for mpc and expected
the content-type header for shoutcast relay and streamtranscoder.
- kh21a. kh20 introduced a bug which could occur with listeners being moved between
- Fix possible stall bug when a source exits and when adding a listener, only
keep the source_tree lock held until a source lock can be acquired.
- tighten up where the certain locks are held/released.
- win32 stat return code fix
- added a few internal checks and some log messages.
- fixes from moo, #1646, status response header first always, #1648, return ICY
response for mpc
- make fh_node use tree for quick lookup
- zero out format structure, remove previous stats for restarting relays
- NULL pointer fix in cases where a relay was being cleared of listeners without
a fallback mount, eg being disabled or removed from slave list or reread XML
- kh20a. crash fix in cases with a relay removed/disabled
- fix possible lock imbalance bug if mutliple workers set
- make sure listeners are not paused if relay is not running
- fix memory leak on each source disconnection.
- fix double free corruption bug if listener_remove used.
- on server shutdown, prevent auth of new listeners only. other auth requests
can continue which allows client details to clean up properly
- relocate where the source count is increased. For source clients it is just
before where it used to be but for relays it is increased before a relay
connection is attempted. Bug in kh19 meant sources kept increasing on relay
- fix possible stalling bug in auth command method.
- listener_connections stat could have a random data bug
- Do not force a master relay recheck on stream start/stop
- build fix on certain platforms.
- small internal code cleanups.
- add 7.xsl processing to present stats for a shoutcast /7.html request, requires
alias of /7.html to /7.xsl, not complete but basics are there
- avl tree of listeners is back in, workers process lists of clients but the tree
enables a quick listener lookup from a source.
- the admin arg require handler left a lock held if not specified.
- make sure the allow/deny/useragent files reload properly
- initialisation bug for auth, more likely to affect win32 gui.
- content-type has to be lower case for shoutcast to recognise the mime type.
- per-mount access log. <accesslog> <name>/file.log</name> ....
- update to the source shutdown process. reduces code in relay shutdown, no point
in duplication. allow listeners to be placed in a waiting state if source is
waiting to start up.
- do not dump stream stats on on-demand relay failure.
- allow for reporting xml parser warnings to stderr at startup
- fix possible auth related corruption with disconnected clients on slow auth
- type fixup on lag calculation on 32bit systems.
- if streamlist fails then drop all slave relays.
- for people who use ICE_MUTEX_ABORT (environment var) to trap for long held
locks, change the name to ICE_LOCK_ABORT
- fix possible memory corruption case in stats and threading
- fix vorbis stream bandwidth increase when certain source clients connect
- YP thread now started from worker as required instead of once per second
- avoid any ::ffff: in IPs
- shutdown order and other internal code cleanups
- use signalfd if available for processing signals in connection thread, saves
waking up the thread.
- make sure the auth mountpoint returned is used instead of the listener
supplied one. Allows for redirecting listeners to alternative mounts
- update command auth, works in a similar way to url auth now.
- stats clients work better under much heavier load. stats notifications
are grouped together if possible before sending, this reduces writes in the
many streams case
- fix possible memory corruption when using multiple workers
- fix possible race case in multiple file handle close
- a few minor timing changes, nothing major.
- kh16b. fix possible crash case with relay failing
- kh16a. fixed a crash bug triggered by flash clients
- you can now fallback to file if the initial mountpont is not available as long
as there is a limit-rate set at some point within the fallback chain.
- A header from url listener add "icecast-auth-user: withintro" indicates that
the response contains intro data to be sent to the this listener. Anything
else indicates the previous behaviour.
- allow source clients to be limited by duration, set by time limit header in
url stream auth
- internal changes to average bitrate calculations. blocks are now removed based
on the timestamp not the block count.
- fix possible race with url auth'd listener getting a 404 response
- try throttle file sending based on duration/bytes sent on file handle and split the
file sending function into two (throttled/unthrottled). reduces memory.
- merge in compile-time allocation counters
- Added some stats for fallback files, and metadata_updated for sources
- minor cleanups for internal API. log message cleanups.
- lock mismatch identified, could cause a stall in the worker.
- relay connection timeout was not being set in all cases
- Do not fallback listeners on relay fail if server shutting down
- minor code cleanups.
- 15a. fix memory corruption when many clients are referring to the same file.
- 15b. fix possible crash on url listener failing.
- Allow stats/listclients/killclient work with file references
- Add timeout parameter to relay. number of seconds for connect to complete.
- fixes for restarting of relays, especially where changes are detected and
where fallback to files were set.
- repeat fallback file
- tuneup client scheduling for file sending
- added a few validation checks, useragent chars, xml config port ranges
- log message cleanups.
- on source exit, listeners do not need to be moved to the file serving engine,
just release any shared data and delete client if not on worker. saves process
work at source exit
- fix possible crash bug with long lock held detection.
- allow listener_add response to include intro content
- merge connection_t into client_t, and shrunk connection_t, reduces memory used
- improve some of the internal client scheduling.
- the reading of the mime types file was done too early
- admin/fallback now updates the mount details if available.
- more source lock cleanups. nothing major
- change the order of the header routines, the format specific routine now calls
the general routine.
- xml based relays could cause a crash on server exit or xml reload
- avoid memory/fd leak on active relay restart.
- cleaner worker thread shutdown.
- win32 update. Prevent Esc closing the GUI. Make the service work better and
is now able to take an alternative xml as parameter. Update libogg in the
build to 1.1.4 and refer to the current directory for dependent DLLs.
- busy looping caused by truncation on 32bit setups.
- avoid possible deadlock between file release and open.
- relay client handles were not being cleaned up fully.
- send Host: header on relay connects.
- big internal thread change. instead of one thread per stream, we now have a fixed
number of worker threads, each processing a set of clients (sources/relays/listeners etc).
- no source or fserve threads now. auth still has thread(s) as the API for libcurl
- worker threads with no clients sleep for long durations.
- defaults to 1 worker thread, overridden by <workers> in <limits>
- listeners get moved to the worker thread running the source, helps caching
- sources get moved to a less busy thread if it would help
- each client has a timestamp to indicate when it wants service. The sleep duration
for the worker is based on the next client to be serviced.
- if fallback to file is specified and the requested mount is not active then a 404
response is returned instead of the file contents.
- fserve engine can handle bandwidth limiting per client. Fallback to file uses this
by using the average incoming bitrate of the source stream as the target limit.
- file serving uses an internal cache of open files, so that 1000 listerners do not
suddenly open 1000 files if a fallback to file occurs.
- move listeners admin request can specify another source or file.
- relay changes
- the client structure is always allocated now, instead of when the relay is started.
- listeners stay on relay if switching to another relay master. Only when they all
have failed does the fallback apply.
- post-kh9 fixes
- fix for possible race with rejected new listeners going through auth
- make mp3-metadata-interval 0 work again
- auth htpasswd checks for details provided like user:pass@host:port/mount and if not
then checks as host:port/mount?user=a&pass=b
- allow shoutcast source client auth work via stream_auth url
- allow for a flash policy file.
- internal cleanup to help merging into trunk
- fix vorbis metadata update via url, broken in recent releases. cleanup some metadata
- drop vorbis stream rebuilding back to flush evert half second, in sync with trunk
- stop false reports of 'not null' in debug logging. no other effect on server
- limit how often the relay mountpoint "already exists" logs.
- minor tweaking to the average bitrate calculation
- remove the internal global time caching, although prevent several time calls if in
- kh9a fixes a race case which could cause a crash with rejected auth'd listeners
- fix memory corruption which affected ogg streams.
- fix for unsupported FBSD/win32 spin locking options. abort on initialisation failure.
- win32 fix from kh7 causing stalling source threads. fallout from kh7 build failure
- merge fix for bug report on trunk for skeleton
- fix for build failure with recent glibc
- fix for admin.cgi metdata updates not applying (kh5 bug).
- minor locking cleanup
- win32 fixes. server restart could cause relay details to become corrupted
- win32 update. updated libxml2/libxslt
- some source stats were not being reported on the xsl pages.
- win32 access log timestamps should now to correct.
- stats fixes, crash case in stats handling, and hidden attribute was not honoured
on xsl pages with recent stats changes.
- if requesting /admin/streams via stats method then allow auth via relay user/pass
or mountpoint auth. receives only stats for slaves
- allow url= tag on metadata update for non-ogg streams, adds StreamUrl.
- fixed possible crash with skeleton stream handling
- actually prevent metadata updates if there is a stream already running from a
- fix build bug on FBSD7
- allow optional queue-len tag in listen-socket, default is 10
- Where's the brown paper bag. boolean settings in the xml were being ignored due
to an incorrect test. update from kh3
- fixed master not informing slaves of streams bug. caused by missed setting in
the kh2 stats update
- more xml checks for badly formed xml files.
- log files can now be in sections not just a single filename ie <errorlog> can
contain <name> <archive> <display> and <level>, accesslog can contain <name>
<ip> and <archive>, playlistlog can contain <name> <archive> <display>. In each
case, name is a filename, archive is a true/false setting for using timestamps
on a reopened file, display is the number of lines to show in the admin page.
- more stats work. Clients are now processed as needed instead of a dedicated
thread per stats client.
- do not allow raw metadata updates if not from the same IP address as the source
client unless it is from the admin user. For some reason some source clients
issue updates even though they are rejected.
- missing lock on streamlist master/slave update, could cause memory corruption.
- update to average bitrate handling. feeback please
- allow strftime expansion on dump filename, applies at the time the stream has
- removed debian directory, it's not maintained really so leave it to the debian
people to deal with it themselves.
- experimental theora keyframe as png patch added. not built by default
- minor memory leak plugged in fserve
- remove stats thread. Stats clients still have their own thread and queue.
- fix low bandwidth theora stream problem.
- failing on-demand relays were not being recovered as they should of.
- encode xml entity data for clients useragent/username
- reduce memory usage in avl nodes, seems to benefit win32 more because of the
unused lock handling
- minor patch for kate detection applied. ogg.k.ogg.k
- minor type cleanup for off_t in fserve
- Added Kate/Skeleton codec handling within Ogg streams. patch by ogg.k.ogg.k
- small changes for stream directory handling, mostly for error cases.
- added /admin/reset?mount=/mountpoint&setting=x to reset stats for that mountpoint.
without setting= all resetable stats are reset. x can specify peak,read or send
or a combination of them (comma deliminated)
- /admin/streamlist.txt uses the same routine as /admin/streams so does not have
a limit on how much is sent back.
- updated win32 code, windows service should be ok again. service name now
includes version to help multiple installs but windows may require a reboot
between removing a service and installing that same service again.
- fix possible crash on reload of xml with bad tags
- small update of web pages
- added per-mount stat for total mbytes sent
- kh34a, avoid possible loop on yp add failure case
- master/slave update.
slave mode only issues a streamlist.txt request if the /admin/streams request
fails. /admin/streams also acts as an mountpoint for the slave relays by
passing a mount= arg and auth.
You can now define a <mount> for /admin/streams to specify how slave
authentication is done eg url (via listener_add) or htpasswd.
Once authenticated, the slave can bypass any limits like max listeners etc.
- The slave also allows for defining a <master> tag block instead of <master-*>
tags allowed are <username> <password> <server> <port> <redirect> <bind>
auth url can take an "icecast-slave: 1" header, which makes client act as a
slave and bypass mount limits.
- build fixups for OpenBSD and cases where IPV6_V6ONLY is missing
- move 'clients' limit check to after admin request check, means admin commands
can work even if the limit is reached. removes a lock taken as well.
- drop connections to known slave relays if we find that the destination source
is a fallback to file, as this would cause high bandwidth usage.
- stats fix for inactive on-demand relay stats without a mount definition.
- added some minor consistency checks.
- crash fix for a failing on-demand relay.
- incorrect logfile settings for timestamp setting, for playlist and access.log
- update for win32 build and initialization.
- hard to trigger but possible race condition on source startup.
- long standing bug identified with many descriptors (eg > 1024 listeners)
- allow YP thread to skip a non-responsive server
- crash fix if no user/pass supplied when stream_auth in use.
- the admin=1 was not passed in POST for stream_auth on metadata updates
- some small code re-organisation in kh28 caused auth htpasswd to fail new listeners
- work around FBSD issue with listening on IPv4/IPv6 sockets
- race fix in yp thread startup.
- manageauth userlist page was blank due to a NULL check, fixed.
- add rejected_mount auth option to redirect unathenticated listeners to an alternative
mountpoint. eg subscription stream redirected to low bitrate
- Allow for url auth to handle 'Mountpoint: ' header for redirecting listeners if they
are to be rejected to an alternative mountpoint on the same server. This is as opposed to
the Location: header which gets a player to redirect, maybe to an alternative server.
- fix potential fserve thread race
- merge listener inline shoutcast metadata into previous block write. It is
very common and saves writing a very short packet per listener at metadata.
- reduce log memory usage when caching lines for web display
- reduce memory usage slightly on 64bit systems, in refbuf/avl storage
- be more 32/64bit clean
- fix possible corruption if reducing qblock size over reload
- get stats thread to update global outgoing_kbitrate stat, no point flooding
the stats queue.
This is recommended over kh25. kh25 introduced a bug that could cause a major resource problem
- fixed accidental bug which caused big problems for non-ogg streams without a
mount defined. With a mount defined the server worked fine.
- if listener_remove used then remove queue reference before auth performed as
queue will expand. race case between auth and source thread on refcounting.
- Add error reporting with above case, makes identification easier.
- internal code cleanups.
- kh24 had a lockup bug on reload if auth url used.
- Don't pass listeners to the authentication queue if no auth handler defined
- auth listener_remove was not triggering if listener_add was not specified.
- make sure avl tree for mime types is cleared.
- remove pthread_exit in fserve, older glibc had a memory leak
- fix memory leak/corruption in YP handler, also make YP thread startup instead
of polling frequently.
- redo server connection lookup. Now we refcount the server connection details
from the xml, make the client refer to it. reduces locking as well
- always call the xml reload from the slave thread.
- fix memory leak over reload, when using multiple master within relay
- incorrect output appended to streamlist output for relays.
- race between stream shutdown and listener_remove could cause memory corruption
with queue buffers. Make source clear less strict.
- cleanup any fserve clients after other threads are down, removes lock race
- reduce burst limiter for listeners on sources.
- if available, log file descriptor limit allowed by kernel.
Main differences from offical icecast release