Upgrading from Debian Jesse 8 to Unstable

As we all know Debian is a very stable Linux distribution which is partly what makes it great, sometimes however you want access to newer packages, while it’s possible to mix stable packages and unstable it often leads to quite a mess, some packages have so many dependencies it’s often easier just to go entirely to unstable.

Unstable despite its name is actually fairly stable for the most part, so upgrading to it isn’t usually a big issue, upgrading is best done as a two step process, first from stable to testing, then testing to unstable, trying to go directly generally will not work except for a freshly installed base system, for you average user I’d recommend stopping at testing and then just install what you need from unstable since package problems can and do occur.

Upgrade Process

First you need to edit /etc/apt/sources.list and change jesse to testing like so, use a mirror closest to you for best performance:

deb ftp://ftp.uk.debian.org/debian/ testing main
deb-src ftp://ftp.uk.debian.org/debian/ testing main
deb ftp://ftp.uk.debian.org/debian/ testing contrib
deb-src ftp://ftp.uk.debian.org/debian/ testing contrib
deb ftp://ftp.uk.debian.org/debian/ testing non-free
deb-src ftp://ftp.uk.debian.org/debian/ testing non-free

Once that is done run the following:

sudo apt-get clean
sudo apt-get update
sudo apt-get dist-upgrade

All being well there should be no package errors here, go ahead and let it upgrade to testing, once it’s done it’s best to reboot and make sure everything is working, in some cases you may have to reinstall your gpu driver.

Once you’re happy everything is working you have two options, you can stay on testing and add the unstable repositories, or dist-upgrade to unstable, to get the latest packages you want, you can use the -t switch with apt-get, aptitude and synaptic to select the target release for example:

sudo apt-get -t testing install some-package
sudo apt-get -t unstable install some-package
sudo synaptic -t unstable

There is also an option in synaptic to set your preferred release.

If you do dist-upgrade to unstable be aware things can break from time to time, mostly packages that are being worked on, fixing this is a simple matter of switching whatever is broken back to the testing version, if you’re not comfortable with doing this stay on testing.

KeePass Password Manager

These days it’s more important than ever that you have a good strong password that’s different for each account you have, keeping track of all these can be quite complicated so having a password manage makes life much easier, you could of course write them down but that has its own risks such as someone reading them or more likely you losing them.

Many people rely on the password managers built in to web browsers, however this is a very bad idea, since there is often no encryption and it can be quite easy to fool the web browser in to giving the stored passwrds which is why I strongly advise people to stop using it.

KeePass

5i7dbt7

Main window

KeePass is a very popular password manager which has been around since 2006, it has many of the features you would expect from such a mature application such as:

  • Encrypted password database
  • Password categories & search
  • Autotype system that doesn’t need browser plugin
  • Password generator
  • Plugins to add more options
  • Free and open source

KeePass was orginally written for Windows but now many ports are available for different operating systems, KeePassX in partiular supports many platforms.

nogljqw

Password generator

There are of course other applications available that do more or less the same, however most are not open source and many depend on cloud based storage which in my opinion could be a security risk since the database is out of your control.

After using KeePass for quite a few years I could never go back to the old way of managing passwords, using this I can use passwords far longer than I could ever be botherd to type which greatly improves security, unfortunately there are still some websites out there that have arbitary limits on password length, PayPal in particular is a good example of this stupidity.

How not to lose your password database

With KeePass which is not cloud based there is always a risk you could lose the password database, one way around this is to have a cloud hosting account with a simple password you can easily remember, it isn’t vital to protect the database since it’s already encrypted, you can then synchronize the database whenever you make changes.

Like anything it’s still a good idea to make periodic offline backups.

Blocking Advertising

Internet advertising is one of the biggest risks to your security and privacy so it’s important to block it if you value these things, some might argue it’s wrong to block advertising but when it puts you and your computer at risk there is no other option.

Internet advertising can be blocked by three main methods:

  • Hosts file
  • DNS filter
  • Browser plugins

Most people these days use browser plugins such as AdBlock, Adblock Plus or my personal preference and recommendation uBlock Origin, these in general do a very good job at blocking advertising but don’t work outside the web browser, they can also be used to remove unwanted elements from a web page giving you a cleaner browsing experience.

Hosts File

The hosts file is a little more complicated to explain, when you go to a website such as http://www.google.com your computer needs to lookup the domain name to obtain the internet address such as 216.58.198.110, this is done by contacting a DNS server which is typically provided by your ISP, however in the early days of the internet there was no DNS servers, instead it looked in a hosts file which manually maps domain names to ip addresses, for example:

216.58.198.110 http://www.google.com

The hosts file typically has priority over the DNS server so you can use it to override the domain name resolution, this is usually done by redirecting the domain to the local loopback address which is 127.0.0.1 or 0.0.0.0, this effectivly blocks the domain.

On windows the hosts file can be found at C:\Windows\System32\drivers\etc\hosts
On Linux and most other UNIX based systems it can be found at /etc/hosts

To make things easier you can find hosts files online that already block the majority of advertising providers and other unsafe domains, I’m currently using Steven Black’s hosts file which is compiled from several different reliable sources.

This applies to all applications on your system but I still recommend it be combined with a browser plugin for maximum coverage.

DNS Filter

Rather than at the hosts file the blocking can also be done at the DNS server level, you can either do this by setting up your own DNS server or by using a public DNS service such as OpenDNS.

Personally I don’t use these public services out of privacy concerns but if you want a very simple method that needs no maintenence this might be for you, one big advantage of this is that it works on devices where you cannot typically access the hosts file.

If you have a Raspberry Pi laying around consider installing pi-hole on it for a super simple hardware DNS server.

Advanced Blocking

Sometimes you may run in to an advert that is not blocked by any of your installed methods, in a web browser it’s easy to add new blocking rules but outside you may need to find which domain it’s coming from.

This is easily done by tools such as Process Explorer or Wireshark which can show all HTTP connections, with a little effort this is usually able to locate the offending domain, for cases where the connection is made directly by IP address you can block it using a firewall such as TinyWall or Windows Firewall.

Foreground Reference Utility

This is a very handy tool I found some time ago that lets you overlay an image on the screen, you can adjust the opacity as well as freeze it so you can manipulate the window below it, similar to a layer in photoshop or any other image editing program.

This is incredibly useful in a number of situations where a method to overlay an image is not available, I often use this in PCB design to verify my dimension are correct when drawing a component footprint.

nhevafr

Once you have the reference image where you want you can lock it by clicking the ‘Overlay’ button, to unlock it again just hit F1.

Download
SHA-1 Checksum: 8aca82bdc28e02493e4364688f6c569cd6600f5b

File Compression Guide

When sending files online good compression is essential for saving bandwidth, while many people these days have quite fast download speed, there are even more that have speeds below 8Mbps.

File compression consists of two parts, the archive and the compression algorithm, many archive formats support various compression algorithms, the most noticeable example of this is the .tar archive, when compressed it’s common practice to add the type of compression as a suffix, for example .tar.gz, .tar.bz2, other formats like .zip, .rar and .7z often specify a preferred compression method.

For this article I’m going to be using 7-zip which offers a variety of compression algorithms and archive types, it’s also completely free and open source.

Testing

This test will be done on three different types of files, the first being the nvidia driver installer (361.91-desktop-win10-64bit-international-whql.exe), the second being a PDF book and the third a large plain text file, this is important since the compression ratio depends on the file type, for instance installers are typically already compressed so I expect minimal compression there.

Files Uncompressed Size
Installer 321 MB (337,507,360 bytes)
 PDF Book 114 MB (120,225,893 bytes)
 Text File  9.13 MB (9,584,473 bytes)

For the first benchmark I will be compressing each with LZMA2 using the 7z archive which is the default and recommended for 7-zip, other options are at defaults, compression level normal, dictionary size 16MB, word size 32, solid block size 2GB, CPU threads 2.

Files Compressed Size Compression Ratio Compression Time
Installer  321 MB  100%  ~43 seconds
PDF  109 MB  95.6%  ~17 seconds
Text  1.40 MB  15.3%  ~4 seconds

As we can see from these results plain text has by far the best compression ratio, while the installer did not benefit at all, in some cases this may actually increase the size, the PDF had a reasonable improvement in size but this is dependent on how the PDF is compressed.

Now let’s try again but with the compression level set to ultra.

Files Compressed Size Compression Ratio Compression Time
Installer
PDF 107 MB 93.8% ~26 Seconds
Text 1.39 MB 15.2%  ~4 seconds

The results of this are rather interesting, the installer caused 7-zip to freeze on ultra so I was unable to see if there is any compression, the PDF shows a reasonable gain at the cost of compression time while the text file remains mostly the same.

Compression level isn’t the only thing you can tweak, dictionary size can have a major effect on the compression ratio but also enormously increases the memory requirement for compression and decompression, the default 16MB is rather conservative, ultra defaults to 64MB which is much better but you can get a little more by increasing it, generally above 128MB gives minimal gain.

This test is a little unrealistic as often you will be compressing many files, let’s try a mix of different file types with an uncompressed size of 132MB

Compression Compressed Size Compression Ratio Compression Time
Default  117 MB  88.6%  ~9 seconds
Ultra  90 MB  68.18%  ~24 seconds
Ultra + 128MB Dict  89.7 MB  67.95%  ~22 seconds

I was a little surprised by these results that a larger dictionary size actually took less time, it really goes to show that the types of files determine how far you can compress more than anything else.

Conclusion

I was expecting more definitive results as to what is better but as these tests show it varies on a case by case basis, I would certainly recommend you stick to LZMA2 as various benchmarks by many people have shown it to be the best in terms of compression ratio, memory and for the most part compression time, things like .zip with deflate (I.E winzip) should be avoided these days.

If you really need good compression then the only true way to do it is to test various settings for what you are trying to compress.

For things like video, audio and images, compression isn’t really the answer, using a different format or codec is the way to go since compression can only go so far.

Useful Free Windows Tools

There are lots of tools that greatly improve the usability of Windows but for one reason or another are often quite obscure.

Process Monitor

vst6ufl

This is an extremely useful tool that allows you to monitor file activity, process activity, registry activity, network activity and more in real time so you can find out exactly what applications are up to.

You can get it here.

Process Explorer

4slhio8

This is a much more useful process manager than the one built in to Windows, aside from a wide range of resource monitors it can show all the loaded dll and other files which can be very handy if you run in to the common problem of being unable to delete a file since it’s open in another process, a lot more information is available as well such as active network connections, threads, GPU usage for example.

You can get it here.

Visual Subst

jusc8d3

This is a graphical interface for the subst command which allows you to map folders to virtual drives, you can of course do it with the command but this is easier, particularly if you make regular changes.

You can get it here.

SuperF4

icon128

One of the most annoying aspects of Windows is how hard it is to kill some full screen applications, with Linux you can almost always switch to a virtual terminal and kill it there but with Windows unless you have a second monitor you’re stuck.

This handy little tool runs taskkill /f on the active full screen application when you hit ctrl + alt + F4, this is far more likely to work than the regular alt + F4 which can be ignored by programs, it also has a feature like xkill which lets you click on the window you wish to kill.

You can get it here.

f.lux

qrh3cfx

Pretty much one of my favourite applications for Windows, it adjusts the color temperature of the monitor at night time to reduce eye strain and give better sleep, after using it for some time I can definitely say it helps, it may seem strange at first but your eyes quickly get used to the more orange color to the point where you don’t even notice it.

You can get it here, it’s also available for Linux and more.