Compressed file archives

There are a few different formats and software packages for archiving files. Some notes about them are here:

Software: Most people have come across “winzip” at some point in the past couple decades. It’s a shareware program for creating zip files that constantly nags you to “upgrade” to the newest version. Many people switched over to “winrar” for it’s extra features, but it’s also shareware. Most operating systems can compress or extract basic zip files now, but a dedicated program can be more effective and feature-full. There are two modern free and open source programs for opening and creating compressed archives: 7zip and Peazip. They both read and write pretty much the same archive types, with some minor feature differences. Interestingly, they both can be installed in “portable” mode on a USB drive so that they can be used on any computer you come across without needing to install anything.

7zip is much more popular and mature of the two, but looks more dated and can be a bit more confusing to use. Due to it’s popularity, there are a bunch of non-official websites providing a download of it. I wouldn’t trust anything but the official site (linked above).

Peazip is newer, and does a good job of being a general file manager as well as archive creator. It has better support for creating encrypted archives and a few extra features like “secure wipe” of files (don’t trust on flash or SSD drives). I would recommend Peazip, but the differences are minor.

 

Formats: Traditionally, the “zip” archive has been commonly used in the Windows environment, while tar.gz files have been used in the L/Unix world. 7zip and Peazip support those legacy formats, but there better ones now. The 7z format (default in both 7zip and Peazip), offers better compression, and optional encryption of contents and filenames (regular zip files can only encrypt contents). The 7z format doesn’t save the complete Unix-type meta-data of files, so on those systems, if it’s important, xz uses the same compression algorithm (LZMA/LZMA2) and can be combined with tar to create tar.xz or txz files with the same compression features as 7z. Note, when saving 7z files, you may want to check that LZMA2, rather than the LZMA method is checked as it adds some extra protection against corruption down the road. Also note that with all compression algorithms, there is a trade-off between compression and speed. Both programs let you select settings from “fastest” to “ultra compression” for any format, depending on what your priority at the time is.

The nice thing about the LZMA algorithm is that the trade-off is generally only for compression. De-compression (or extraction) is about the same speed regardless of the setting and is as fast as the poorer compression algorithms like gzip and zip. This is great for things that will be, say, posted online and downloaded by many people. Spending a little extra time compressing the file before posting it will save a bunch of bandwidth, but won’t effect the people who download and extract it.

For multimedia, general data compression algorithms really suck. That’s why we have MP3, OGG/Theora, and it’s amazing successor Opus for compressing audio, and h.264, etc for video content. They throw away much of the information in the file that we don’t really hear or see in order to reduce file size, and so are considered “lossy” formats. If you want to compress audio *without* sacrificing that information and maintaining perfect copies, you still don’t want to “zip” your uncompressed audio files or CDs, it’ just won’t do much. Instead, you need a program that can rip or encode to the FLAC lossless audio codec. It’s basically the equivalent of zip/7z/xz for audio, compressing the file size considerably, without changing a single bit of information.

Geek note: Lzo is a really neat modern compression algorithm that does rather poorly at compressing data, but is incredibly fast (especially at decompression). Rather than for archives, it is great for things like disk and memory compression, modern CPUs can decompress the data faster than it can be read from a disk, effectively increasing I/O speed.

Komodo is a great go-to text editor

I hate IDEs and text editors that have text everywhere in buttons and menus. When you are writing, and have a wall of text in front of you, there is no benefit in being distracted by extra, unnecessary text everywhere. Now, there are “zen-like” editors that just edit text full-screen with no distractions, but they tend to be highly limited in functionality. Vi(m) may be great, but I want to be able to use my mouse without thinking when I want to edit text, and would like advanced functions to be available in clickable menus at the drop of a hat. Komodo Edit an open source text editor that is based on the firefox gui and can scale down to an incredibly attractive, minimal text editor, and up to nearly an IDE in functionality in a few mouse clicks. It’s absolutely great. Unsurprisingly, being based of firefox, it’s not exactly light on memory. What matters to me, though, is that it opens nearly instantly and It’s UI is very snappy subjectively.

I set it to hide the main menu (it put’s it in a “hamburger menu” on the toolbar like firefox), and removed all but a neat, minimal set of icons on the toolbar. One of which pops open and closes a sidebar that let’s me navigate projects and open files, so I don’t need a tab bar.  When I open a random text file, it opens quickly and presents an incredibly simple text editor by default. When working on a project, I can pop open the project sidebar, split the text editor to view two files at once, enable the minimap scroll bar, set up macros, run commands on text and view output, etc. It quickly scales up and down while remaining clean on the screen and un-distracting.

The only major issue I have had is that I would like to add the Sciviews-K addon that gives it lot’s of functionality for the R programming language. I use Rstudio a lot for R, but it would be nice to have that functionality in my general purpose go-to editor as well. The released plugin is only set for up to version 6 of Komodo, and even if I change the “maxversion” string in the plugin, it doesn’t work (It installs, and R menus are available, but they do nothing. Also no syntax highlighting). If I build the plugin from SVN, it installs, but doesn’t make any changes whatsoever to the editor. Unfortunately, Komodo doesn’t include R (or S) in it’s default syntax highlighting capability. I can do without all the R functionality, but I can’t deal without syntax highlighting. Fortunately, setting the default highlighting for R source files to matlab does a surprisingly decent job. I’ll just leave it at that for now. If I need to quickly edit a R file, I can open it and work with it comfortably with Komodo, but when doing more involved work, I’ll stick with Rstudio for all it’s advanced IDE functionality.

A word of note. Komodo Editor is an open source project hosted on github. Unfortunately, it doesn’t seem to be packaged by any linux distributions, so you will have to install it from the package on their website or build it. The binary package they host is a tar archive with a shell script installer that installs the editor in your home directory, so you may need to add the binary to your path. Unfortunately, this binary package has an additional, non-free license, so if you are concerned, you may want to build it from source. They are unclear as to what was added to the binary installer that you may be missing by building from source. Hopefully it will become more popular and built by distributions soon. Also note that the developer also sells a proprietary, commercial project, Komodo IDE, that is built on the Editor base and their website and the text editor’s “start page” (which can be disabled), advertises to upgrade to the IDE. Since the start page can (and should) be disabled, I wouldn’t classify this as adware, but some people may be uncomfortable with this. You are forewarned.

Mumble voice and chat.

There has been lots of talk, due to recent events, about voice and text chat that is respectful of privacy. Most of these projects use cutting-edge end-to-end encryption schemes that are interesting, but are very much experimental designs and not exactly easy or reliable. Recently, I’ve been playing around with a chat program called Mumble that is popular for gaming.  Mumble is a fairly traditional server-based group voice and text chat system. While, like Facebook, Skype, and Google hangouts, you have to trust the server, unlike those services, the Mumble server is designed to be brain-dead simple to host your own. It runs on various operating systems, with almost no configuration, and requires at most a single forwarded port on your home router. It’s a really neat package, and even though I’m not a gamer, I can see various neat uses for it.

 

The software is broken up into too parts. The Mumble server basically provides the service, like Google or Facebook does, but can run on your own machine. The client is a program or app that runs on your desktop, laptop, tablet, or phone and connects to the server to chat with others on it. A server can have different “rooms” so that groups can divide up if necessary. A “room” has both a group text and voice chat. You can also click on a user to send them a direct text message.

 

Some really nice features:

• Lot’s of “Public” servers hosted by volunteers that can be connected to quickly.
• Hosting your own server on a personal computer, home server, or VPS is easier than any other server I have ever seen. Mumble is really designed for easy self-hosting by people that don’t normally self-host stuff.
• Uses the fantastic Opus codec by default! Opus is a new exciting audio codec (like Mp3) for both voice and music that packs an unreasonably great quality signal into a fraction of the bandwidth of older codecs.
• Most of the “administration,” like creating new “rooms,” and other settings is done in the client software by users, so the server rarely needs to be touched.
• Clients for everything, Linux, Windows, Android, IOS, etc.
• Push-to-talk, continuous audio, and voice-activated options for various uses (in client).
• Everything, signalling, text chat, voice, is always encrypted too and from the server. The clients are designed to *not* scare users away when self-signed certificates are used, but warn them if it changes.
• Server uses extremely minimal resources, and due to the Opus codec, can be run on a basic home DSL connection for moderate sized groups.
• Clients can be configured to trigger push-to-talk with various keys, buttons, etc.

Reading around the internet, I have found people using Mumble as a private “intercom” between offices, or family member’s phones, or on a local WiFi for organizing a group of people. Because the server can be run on a local network, it can be used without an internet connection entirely. My plan is to use it for local family chat. All of the family’s computers are set to automatically connect to my home Mumble server, either through the local network or, in the case of laptops, over the internet, self-minimize, and trigger the push-to-talk with the bottom left two keys (Ctrl+Shift). Basically, this makes a handy intercom that can be used without a second thought. I have also added the apps to my phone, and a tablet, so they can be used as like private, mobile, “radios.” I would like to build a hardware intercom (i.e., speaker in a box with a button on the wall), connected to a pi or a computer on the network for the kitchen. It could be dead simple, with the button connecting/disconnecting the microphone and the client set to voice-activation, so only 3 wires are needed, and no need to hack up some sort of IO for the push-to-talk. It would also be a neat way to collaborate with colleagues when combined with an Etherpad instance or the Gobby collaborative text editor.

 

Mumble isn’t perfect. There are a few issues. The clients for Android and IOS could use some polish. They may occasionally disconnect without notifying the user. The devices themselves don’t have a good way of providing a push-to-talk button without opening the app (I haven’t tried using a button on a headset). Switching between push-to-talk and a more continuous talk mode requires going into the client settings. Generally, though, it works well, and the developers are working hard and receptive to suggestions. The software is completely Free and Open Source, so you can look it over and get involved in bug reports, translations, or coding.

 

 

Audiophile wireless headphones for free

Here’s a neat idea. MPD (Music Player Daemon) has added opus streaming support. Opus allows absolutely excellent quality audio to be compressed down to a stream size that would be considered “low quality” for mp3. BSPlayer for Android has the ability to play opus streams. I compiled and installed the latest version of MPD on my internet facing workstation, disabled remote access to mpd control, and created a http opus stream for output. With a Mpd client on my workstation to control the music playing, and BSPlayer connected to the http stream, I have nearly CD quality audio on my wired headphones (connected to my Android phone), playing music from (and controlled by) my desktop. Yeah, it’s not really wireless, but it’s a lot easier to slip a phone into my pocket and walk away than it is to do that with my desktop computer. The neat thing, is, the “range” of my “wireless headphones” is basically infinite as it goes over the internet.

Configuration notes:

• Opus-tools and opus-tools-dev are installed so that mpd is compiled with opus streaming support.

• libsamplerate dev package needs to be installed so it can be used for sample converting between 44.1 (CD audio) and 48Khz (soundcard audio). Without it, sound quality was terrible, with aliasing and artifacts.

• be sure to check the output when running ./configure. Things with a “-” sign rather than a “+” sign are not being compiled in due to lack of dev packages.
• Point BSPlayer to http://machine_address:8000 and ensure that the network buffer (in settings) is on high.

Here are the necessary config lines from mpd.conf:

# Only allow mpd control connections on local machine (or SSH tunnel)
bind_to_address "localhost"

# Opus http output (In my own testing, I cannot tell 128kb opus from CD audio, on 96kb,
#                 I could hear slight distortion on complex music (orchestral strings especially)
audio_output {
        type            "httpd"
        name            "Opus shout stream"
        encoder        "opus" 
        port        "8000"
#       bind_to_address    "0.0.0.0"        # optional, IPv4 or IPv6
#       quality        "5.0"            # do not define if bitrate is defined
        bitrate        "128000"            # do not define if quality is defined
#       format   "48000:16:2" #Disable to use input file format
        max_clients    "2"            # optional 0=no limit
}

# Use a very good algorithm to convert CD audio to Soundcard audio sample rates. 
# See mpd.conf man page
samplerate_converter        "0" # MPD's best, most CPU intensive algorithm.

Opus Audio Codec

Opus is a relatively new audio codec (like Mp3). It’s Free, open source, part of future web standards, and is astoundingly good at compressing audio down into small file sizes while maintaining good sound quality. Lots of software plays it (Firefox, Chrome, any audio player on Linux that uses Gstreamer, etc). Unfortunately, most of the CD ripping and audio converting programs don’t yet do so. Also, on Android, as far as I know, only Bsplayer will play it so far (though cSipSimple can use it as a VOIP codec!).

Anyway, here is a  script I wrote that simplifies using ffmpeg’s ‘avconv’ and the opus-tools package to convert virtually any file to opus. I have used this for converting long lecture recordings down to a small filesize and cramming more music on my limited phone space:

https://github.com/JasonLocklin/bin/blob/master/opusconv

Digital TV antenna in Kitchener-Waterloo

It’s been a little over a year since Canadian broadcasters switched from the blurry, static filled old analogue TV system to the modern, digital broadcast standard known as ATSC. Public “information” advertisements have convinced many people that the days of putting up an antenna and getting free TV are over, or somehow illegal. This couldn’t be further from the truth! The new digital TV (DTV) standard means high quality, high definition TV without static or blurriness. That is, if you can get it. Some places, like Toronto or Hamilton, have dozens of strong signals. That means people in those areas can use cheap antennas to pick up every Canadian and American channel for free. Here in Kitchener-Waterloo, however, bringing in more than a few channels requires some work.

Myths about Digital “Over The Air” TV

1. “OTA doesn’t exist anymore”
   Most broadcasters in Canada are owned by cable/satellite companies, so they don’t explicitly advertise that their channels are available for free over the air, but they do broadcast all the primary networks in Canada entirely for residential reception -and they collect advertising dollars to pay for it.
2. “OTA TV is illegal”
   This seems to come from confusion with the grey market satellite that used to be common in Canada. Basically, people purchased Free to Air (FTA) satellite equipment designed for the *Legal* use of  watching  the FTA channels available on various satellites (Mostly international and religious channels). They would then *illegally* use that equipment to decode encrypted channels intended for pay services only. This is entirely unrelated to over the air TV (broadcast from large towers near cities), which has always been free and legal in Canada.
3. “You need a digital antenna”
   Antennas are designed for frequencies, not what is carried on that frequency. As always in Canada, over the air TV is broadcast on VHF and UHF channels.   1-13 are VHF and the rest are UHF. Many areas do not have VHF channels, so you may only need a UHF only antenna. The classic indoor “rabbit ear” works for VHF and the “loop” type wire antennas work for UHF. Outdoor antennas should also be selected based on whether you need UHF, VHF, or Both. Most of those plastic covered “digital” antennas are garbage. If the antenna claims more than about 15db of gain, they are using a built-in preamp and are using that to “fudge” the numbers. If they do this I can guarantee the antenna is not worth your money. You can consult this forum for a list of the best antennas available.
4. “You need a digital converter box”
   Only to use an old TV. Most new TVs have digital tuners. If your remote control has a decimal button (i.e., for entering channel 13.1), then it has a digital tuner.  You can also use a tuner card in a computer rather than a converter box.
5. “You need 100+ channels to make TV watchable, OTA only gives a handful”
   Half true. OTA will give you at most a few dozen channels. In KW, I “only” get 11-12. However, on cable, many of the channels just duplicate content or show it at different times. Take the money you save on a subscription cost and buy or build a digital PVR/DVR. A PVR makes each channel 24 times more useful :-).

First, check the predicted signal strengths from TVfool using my coordinates here. You can use Google maps to find your coordinates. Locate your house, right click on the location of your house and select “What’s here?” The new search term becomes the coordinates. The output is kind of technical, but basically channels in green can be received with something indoor, yellow and red channels require an outdoor antenna and the rest are just there to tease you. Googling the callsign can give you more info about which channel is actually which. Note the “Real” channel tells you whether you need a UHF or VHF antenna to receive a particular channel (<14 is VHF).

I learned most of what I know about OTA on the Digital Home Forum.

My Setup in Kitchener-Waterlo

In KW, TVO, Global, and CTV are local and can be picked up by any old piece of wire. The local VHF channels (Global and CTV) are so strong it doesn’t really matter whats used. Stations from Toronto (CBC, french CBC, CityTV, Omni 1 and 2) are UHF, so in KW, you can get most of the channels via a UHF only antenna pointed at Toronto. Depending on how high your house is, you may be able to get away with a small antenna on a wall. I’m low, so I have a big UHF beam antenna on the roof. For some lucky locations in KW, such as those at higher elevation with SSW view, Buffalo stations are receivable with a rooftop antenna. I have no chance at many of these American networks, but occasionally can catch a glimpse of PBS.

My antenna is an Antenna’s Direct 91XG mounted on a 5 foot tripod on my roof. This is basically the highest gain (i.e., strongest, most directional) UHF antenna available. It’s on a rotor so that I can swing it over to point to Buffalo, or Hamilton -though I have found that I don’t use it enough to need it. I have a CM7777 preamp up there. The preamp basically amplifies the signal (and the noise) so that the signal loss along the long coax down to the basement is negligible. Preamps are only useful when you have a long cable like I do. The coax is connected to a grounding block before entering the basement. It is connected to the house ground outside to avoid static or minor surges from coming into the house and affecting my equipment. I have an HDhomerun tuner connected. This provides two digital tuners on my home Ethernet for my networked PVR to record from.

Reliable Channels:

• CTV Baden
• TVO Kitchener (also TVO Toronto, but it’s the same as Kitchener)
• CBC Toronto
• CityTV Toronto
• Omni 1 Toronto
• Omni 2 Toronto
• CBC (French) Toronto
• Global Toronto
• CTS Hamilton

Unreliable Channels:

• Global Toronto (Ch 6 repeater in Kitchener)
• CW Buffalo
• CHCH Hamilton

Todo:

So far, I am not entirely happy with the setup. The VHF channels sometimes break up severely and I think it has something to do with the very strong signal overloading the preamp. The Toronto UHF signals occasionally also break up somewhat. I would also like to pick up CTV2 from London. It would also be nice if CHCH London was more reliable -it has a strong signal, but my UHF antenna kills it most of the time.

I have purchased a UHF/VHF duplexor, so I can connect a separate VHF antenna *after* the preamp (critically, the duplexor passes power for the preamp to the UHF side). This way only the weak UHF signals will be amplified. The CM7777 also has a built in duplexor, so I can use that to totally disable all VHF input to the preamp and devote all of it’s power to UHF (perhaps making them more reliable as well). By permanently mounting the VHF antenna pointing toward London, CTV, Global, CHCH Hamilton and CTV2 London should be always strong and clear, regardless of where the UHF antenna is pointing. This is the main benefit to having a separate antennas in KW -a single VHF/UHF antenna would not be able to pick up Toronto and Hamilton/London at the same time. Unfortunately, CHCH is supposed to switch to UHF at some point in the future, at that point I might have to swing the UHF around to receive it depending on it’s strength.

CTV2 London is receivable with a simple dipole (rabbit ears) at my antenna height according to tvfool, however, without a preamp, I need several db of gain to overcome the line loss getting to my tuner. Furthermore, I don’t want to build something and find out it is too knife-edge to work perfectly all the time. So I’m building a Yagi antenna specifically for CH10. Yagi’s are basic antennas that are relatively easy to build, small-ish, very effective, but really only good for one channel. Fortunately, all the other channels, including CHCH london, are quite strong so any VHF Yagi will pick them up just fine regardless of it’s design frequency.

Homebrew VHF antenna

I have settled on building one based on VK5DJ’s Yagi calculator with 2 directors and 1 reflector. That gives me an antenna approximately  2.5 feet long and about that wide, with 6.5db of gain over a dipole. Calculating all of the losses amongst the coax and various connections, I should have at least 4db of surplus signal (i.e, more than double the signal strength needed at the tuner). Of course this also gives a little bit of wiggle room if my antenna is not perfect. Hopefully this will be enough to have 100% reliable signal from London, and avoid all the current problems with the strong VHF channels. Fortunately we have a Metal Supermarkets here in Kitchener, so I have some very nice pieces of aluminium for the boom, reflector and directors, and a nice heavy piece of copper ribbon for the driven element. I will make further posts here when I build it, including pictures.

Also, I would also like to make a single point ground on the inside like this.