Compressed file archives

There are a few different formats and software packages for archiving files. Some notes about them are here:

Software: Most people have come across “winzip” at some point in the past couple decades. It’s a shareware program for creating zip files that constantly nags you to “upgrade” to the newest version. Many people switched over to “winrar” for it’s extra features, but it’s also shareware. Most operating systems can compress or extract basic zip files now, but a dedicated program can be more effective and feature-full. There are two modern free and open source programs for opening and creating compressed archives: 7zip and Peazip. They both read and write pretty much the same archive types, with some minor feature differences. Interestingly, they both can be installed in “portable” mode on a USB drive so that they can be used on any computer you come across without needing to install anything.

7zip is much more popular and mature of the two, but looks more dated and can be a bit more confusing to use. Due to it’s popularity, there are a bunch of non-official websites providing a download of it. I wouldn’t trust anything but the official site (linked above).

Peazip is newer, and does a good job of being a general file manager as well as archive creator. It has better support for creating encrypted archives and a few extra features like “secure wipe” of files (don’t trust on flash or SSD drives). I would recommend Peazip, but the differences are minor.

 

Formats: Traditionally, the “zip” archive has been commonly used in the Windows environment, while tar.gz files have been used in the L/Unix world. 7zip and Peazip support those legacy formats, but there better ones now. The 7z format (default in both 7zip and Peazip), offers better compression, and optional encryption of contents and filenames (regular zip files can only encrypt contents). The 7z format doesn’t save the complete Unix-type meta-data of files, so on those systems, if it’s important, xz uses the same compression algorithm (LZMA/LZMA2) and can be combined with tar to create tar.xz or txz files with the same compression features as 7z. Note, when saving 7z files, you may want to check that LZMA2, rather than the LZMA method is checked as it adds some extra protection against corruption down the road. Also note that with all compression algorithms, there is a trade-off between compression and speed. Both programs let you select settings from “fastest” to “ultra compression” for any format, depending on what your priority at the time is.

The nice thing about the LZMA algorithm is that the trade-off is generally only for compression. De-compression (or extraction) is about the same speed regardless of the setting and is as fast as the poorer compression algorithms like gzip and zip. This is great for things that will be, say, posted online and downloaded by many people. Spending a little extra time compressing the file before posting it will save a bunch of bandwidth, but won’t effect the people who download and extract it.

For multimedia, general data compression algorithms really suck. That’s why we have MP3, OGG/Theora, and it’s amazing successor Opus for compressing audio, and h.264, etc for video content. They throw away much of the information in the file that we don’t really hear or see in order to reduce file size, and so are considered “lossy” formats. If you want to compress audio *without* sacrificing that information and maintaining perfect copies, you still don’t want to “zip” your uncompressed audio files or CDs, it’ just won’t do much. Instead, you need a program that can rip or encode to the FLAC lossless audio codec. It’s basically the equivalent of zip/7z/xz for audio, compressing the file size considerably, without changing a single bit of information.

Geek note: Lzo is a really neat modern compression algorithm that does rather poorly at compressing data, but is incredibly fast (especially at decompression). Rather than for archives, it is great for things like disk and memory compression, modern CPUs can decompress the data faster than it can be read from a disk, effectively increasing I/O speed.

Audiophile wireless headphones for free

Here’s a neat idea. MPD (Music Player Daemon) has added opus streaming support. Opus allows absolutely excellent quality audio to be compressed down to a stream size that would be considered “low quality” for mp3. BSPlayer for Android has the ability to play opus streams. I compiled and installed the latest version of MPD on my internet facing workstation, disabled remote access to mpd control, and created a http opus stream for output. With a Mpd client on my workstation to control the music playing, and BSPlayer connected to the http stream, I have nearly CD quality audio on my wired headphones (connected to my Android phone), playing music from (and controlled by) my desktop. Yeah, it’s not really wireless, but it’s a lot easier to slip a phone into my pocket and walk away than it is to do that with my desktop computer. The neat thing, is, the “range” of my “wireless headphones” is basically infinite as it goes over the internet.

Configuration notes:

• Opus-tools and opus-tools-dev are installed so that mpd is compiled with opus streaming support.

• libsamplerate dev package needs to be installed so it can be used for sample converting between 44.1 (CD audio) and 48Khz (soundcard audio). Without it, sound quality was terrible, with aliasing and artifacts.

• be sure to check the output when running ./configure. Things with a “-” sign rather than a “+” sign are not being compiled in due to lack of dev packages.
• Point BSPlayer to http://machine_address:8000 and ensure that the network buffer (in settings) is on high.

Here are the necessary config lines from mpd.conf:

# Only allow mpd control connections on local machine (or SSH tunnel)
bind_to_address "localhost"

# Opus http output (In my own testing, I cannot tell 128kb opus from CD audio, on 96kb,
#                 I could hear slight distortion on complex music (orchestral strings especially)
audio_output {
        type            "httpd"
        name            "Opus shout stream"
        encoder        "opus" 
        port        "8000"
#       bind_to_address    "0.0.0.0"        # optional, IPv4 or IPv6
#       quality        "5.0"            # do not define if bitrate is defined
        bitrate        "128000"            # do not define if quality is defined
#       format   "48000:16:2" #Disable to use input file format
        max_clients    "2"            # optional 0=no limit
}

# Use a very good algorithm to convert CD audio to Soundcard audio sample rates. 
# See mpd.conf man page
samplerate_converter        "0" # MPD's best, most CPU intensive algorithm.

Opus Audio Codec

Opus is a relatively new audio codec (like Mp3). It’s Free, open source, part of future web standards, and is astoundingly good at compressing audio down into small file sizes while maintaining good sound quality. Lots of software plays it (Firefox, Chrome, any audio player on Linux that uses Gstreamer, etc). Unfortunately, most of the CD ripping and audio converting programs don’t yet do so. Also, on Android, as far as I know, only Bsplayer will play it so far (though cSipSimple can use it as a VOIP codec!).

Anyway, here is a  script I wrote that simplifies using ffmpeg’s ‘avconv’ and the opus-tools package to convert virtually any file to opus. I have used this for converting long lecture recordings down to a small filesize and cramming more music on my limited phone space:

https://github.com/JasonLocklin/bin/blob/master/opusconv