Centrino Ultimate N-6300 slow to connect to Wi-Fi network

Wi-Fi connections from my laptop became unstable after I upgraded it to Ubuntu 19.04. To get them to work, I had to turn the Wi-Fi adapter on and off several times, which quickly became annoying.

The following attempts to fix the problem were unsuccessful:

  • disabling IPv6
  • disabling power-saving on the Wi-Fi adapter
  • setting bt_coex_active=0 during kernel loading to disable Bluetooth/Wi-Fi interference protection

Eventually the correct solution was to disable 802.11n with the following procedure:

  1. Edit or create a configuration file under /etc/modprobe.d (e.g. iwlwifi_custom.conf).
  2. Add this line and save the file.
    options iwlwifi 11n_disable=1
  3. Reload the iwlwifi module.
    sudo rmmod iwldvm
    sudo rmmod iwlwifi
    sudo modprobe iwlwifi
  4. Connect to the Wi-Fi network.

SO #57225262—Inputting list of passwords in a class

Stack Overflow question ‘Inputting list of passwords in a class’ is more interesting as an object-oriented design (OOD) exercise than as a debugging exercise, but it also provides an opportunity to practise some Python programming.

The requirements identified in the problem are:

  • represent a set of credentials comprised of a username and a password;
  • represent a user account made up of an identifier and one active set of credentials and zero or more inactive credentials;
  • store user accounts that can be retrieved subsequently by username;
  • validate a given set of credentials, using the stored user accounts.

Additionally, a user account has the following constraints:

  • It must be uniquely identifiable by the username of the active set of credentials.
  • A set of credentials must be used only once for a given user account.

The design model consists of multiple classes, each addressing a single responsibility as follows.

ClassResponsibility
Credentialsrepresent a set of credentials
UserAccountrepresent a user account
AccountStorestore user accounts, enforcing the uniqueness constraint
Authenticationvalidate supplied credentials

The program code implements the model very faithfully and uses the same class names, thus making it easy to reason about. It has been tested with Python 2.7.15rc1 and has been checked for PEP8 compliance.

NOTES

For brevity, the solution presented here does not:

  • implement best practices used for security in real-world scenarios, such as salting and hashing;
  • enforce uniqueness of user account identifiers;
  • consider performance factors.

In lieu of conventional unit tests, a main routine exercises use-cases to ensure that the program works properly.

class Credentials:

    def __init__(self, username, password):
        self.username = username
        self.password = password

    def has_username(self, username):
        return self.username == username

    def matches(self, credentials):
        return self.username == credentials.username and \
            self.password == credentials.password

class UserAccount:

    def __init__(self, user_id):
        self.user_id = user_id
        self.active_credentials = None
        self.past_credentials = []

    def add(self, credentials):
        self._check_uniqueness(credentials)
        if self.active_credentials is None:
            self.active_credentials = credentials
        else:
            self.past_credentials.append(self.active_credentials)
            self.active_credentials = credentials

    def has_username(self, username):
        return self.active_credentials.has_username(username)

    def has_same_username(self, user_account):
        return self.has_username(user_account.active_credentials.username)

    def has_credentials(self, credentials):
        return self.active_credentials is not None and \
            self.active_credentials.matches(credentials)

    def _check_uniqueness(self, credentials):
        if self.has_credentials(credentials):
            raise Exception('These credentials are currently in use.')
        for c in self.past_credentials:
            if c.matches(credentials):
                raise Exception(
                        'These credentials have been used in the past.')

class AccountStore:

    def __init__(self):
        self.user_accounts = []

    def add(self, user_account):
        self._check_uniqueness(user_account)
        self.user_accounts.append(user_account)

    def find_by_username(self, username):
        for ua in self.user_accounts:
            if ua.has_username(username):
                return ua
        return None

    def _check_uniqueness(self, user_account):
        for ua in self.user_accounts:
            if ua.has_same_username(user_account):
                raise Exception(
                        'An account with the same username is already active.')

class Authentication:

    def __init__(self, account_store):
        self.account_store = account_store

    def validate(self, credentials):
        user_account = self.account_store.find_by_username(
                credentials.username)
        if user_account is None:
            return False
        return user_account.has_credentials(credentials)

if __name__ == '__main__':
    credentials = Credentials('user1', 'password1')
    user_account = UserAccount(101)
    user_account.add(credentials)

    account_store = AccountStore()
    account_store.add(user_account)

    user_account1 = account_store.find_by_username('user1')
    print 'user_account1', user_account1

    user_account2 = account_store.find_by_username('user2')
    print 'user_account2', user_account2

    authentication = Authentication(account_store)
    print 'Expecting True...', authentication.validate(
            Credentials('user1', 'password1'))
    print 'Expecting False...', authentication.validate(
            Credentials('user2', 'password1'))
    print 'Expecting False...', authentication.validate(
            Credentials('user1', 'password2'))

    user_account.add(Credentials('user1', 'password2'))
    print 'Expecting True...', authentication.validate(
            Credentials('user1', 'password2'))
    print 'Expecting False...', authentication.validate(
            Credentials('user1', 'password1'))

    try:
        user_account.add(Credentials('user1', 'password1'))
    except Exception:
        print 'Expecting exception... Pass'

    try:
        user_account.add(Credentials('user2', 'password1'))
        print 'Not expecting exception... Pass'
        print 'Expecting True...', authentication.validate(
                Credentials('user2', 'password1'))
    except Exception:
        print 'Not expecting exception... Fail'

    try:
        user_account1 = UserAccount(102)
        user_account1.add(Credentials('user1', 'whatever'))
        account_store.add(user_account1)
        print 'Expecting True...', authentication.validate(
                Credentials('user1', 'whatever'))
    except Exception:
        print 'Not expecting exception... Fail'

    try:
        user_account2 = UserAccount(103)
        user_account2.add(Credentials('user1', 'whatever'))
        account_store.add(user_account1)
        print 'Expecting exception... Fail'
    except Exception:
        print 'Expecting exception... Pass'


The output of the program is:

EY@LENNY:~/Source/junk/python/pwman$ python all.py
user_account1 <__main__.UserAccount instance at 0x7faa36f11170>
user_account2 None
Expecting True... True
Expecting False... False
Expecting False... False
Expecting True... True
Expecting False... False
Expecting exception... Pass
Not expecting exception... Pass
Expecting True... True
Expecting True... True
Expecting exception... Pass
EY@LENNY:~/Source/junk/python/pwman$

Programming Arduino in Atmel Studio

As a fun way to improve my C, I started programming the Arduino using Atmel Studio instead of the friendlier Arduino IDE.

Below is my very first program. It blinks a red LED and a white LED according to a preset pattern.

#define F_CPU 16000000UL

#include <avr/io.h>
#include <util/delay.h>

struct Led
{
    int pin;
    int blinks;
    int on_duration;
    int off_duration;
};

void delay_ms(int ms)
{
    while (ms-- > 0) {
        _delay_ms(1);
    }
}

void blink(const struct Led* const led)
{
    for (int i = 0; i < led->blinks; i++) {
        PORTB |= led->pin;
        delay_ms(led->on_duration);
        PORTB &= ~led->pin;
        delay_ms(led->off_duration);
    }
}

int main(void)
{
    const struct Led white_led = { 1<<PB6, 10, 100, 500 };
    const struct Led red_led = { 1<<PB7, 10, 500, 1000 };

    DDRB |= white_led.pin;
    DDRB |= red_led.pin;
    while (1) {
        blink(&white_led);
        blink(&red_led);
    }
}

With C in Atmel Studio and the AVR-LIBC library, IO ports are manipulated by changing bit patterns in the registers associated with the relevant ports. This requires a good understanding of bitwise operations in C despite the availability of macros to simplify the task.

For example, to set pin 12 of the Arduino to output mode, bit 6 of register DDRB must be set to 1. To do so requires an | (OR) operation with a bit pattern operand where bit 6 is set to 1 and the rest set to 0, so that the states of the other bits in the register are not disturbed.

Using macros DDRB and PB6 defined in AVR-LIBC, this is done like this: DDRB |= 1 << PB6 .

If you are new to C and are unfamiliar with macros, you might wonder about that statement. Besides, DDRB and PB6 are not referenced anywhere else in my program, so how does this line of code work?

DDRB is a macro that expands into C code to dereference the address of the register associated with setting the mode for pin 12, and PB6 is just a symbolic constant for the value 6. In the statement above, by shifting the value 1 left by 6 positions, we create a new value which is then applied to the bit pattern stored at the dereferenced address with an | operation to turn bit 6 of the register to 1. In this case, this sets pin 12 to output mode.

In a nutshell, the sequence of operations is as follows.

Step 1:

1 << 6 = 01000000

Step 2:

Assuming the register DDRB is initially 00000001:

00000001 | 01000000 = 01000001

In my C program, the result of step 1 is assigned to struct field Led.pin and is used as the second operand for the operation in step 2.

It took about an hour to refresh my knowledge of bitwise operations, but the real challenge was interpreting the Arduino schema and the information in datasheets, especially to find the right registers to manipulate.

Hacking SSL support into smtpop.dll

We use smtpop.dll in one application to retrieve email from a POP3 mailbox. Today we had to connect to a mailbox over SSL, which smtpop.dll cannot do.

Our code to retrieve email is in a façade class, so I expected a simple replacement of smtpop.dll with another library and calling new methods. But the tight coupling of the façade with the interface of smtpop.dll classes meant that we needed something that exposed the exact same interface.

After trying several things, I decompiled smtpop.dll and created a new class with the interface that we needed. I was lucky that only two methods, Open and Quit, needed to be changed.

namespace ClassLibrary4
{
    using System.IO;
    using System.Net.Security;
    using System.Net.Sockets;

    using SmtPop;

    public class Pop3ClientWithSsl : POP3Client
    {
        #region Fields

        private SslStream sslStream;

        #endregion

        #region Constructors and Destructors

        public Pop3ClientWithSsl()
        {
            this.UseSsl = true;
        }

        #endregion

        #region Public Properties

        public bool UseSsl { get; set; }

        #endregion

        #region Public Methods and Operators

        public new int Open(string hostname, int port, string username, string password)
        {
            if (this.UseSsl)
            {
                return this.OpenWithSsl(hostname, port, username, password);
            }

            return base.Open(hostname, port, username, password);
        }

        public new string Quit()
        {
            try
            {
                return base.Quit();
            }
            finally
            {
                this.m_streamReader.Close();
                this.m_streamWriter.Close();
                if (this.UseSsl)
                {
                    this.sslStream.Close();
                }
            }
        }

        #endregion

        #region Methods

        private int OpenWithSsl(string hostname, int port, string username, string password)
        {
            this.m_host = hostname;
            this.m_port = port;
            this.m_user = username;
            this.m_tcpClient = new TcpClient(hostname, port);

            this.m_netStream = this.m_tcpClient.GetStream();
            this.sslStream = new SslStream(this.m_netStream, false);
            this.sslStream.AuthenticateAsClient(hostname);

            this.m_streamReader = new StreamReader(this.sslStream);
            this.m_streamWriter = new StreamWriter(this.sslStream) { AutoFlush = true };

            string welcome = this.m_streamReader.ReadLine();
            if (welcome != null && welcome.StartsWith(+OK))
            {
                return this.SendLogin(username, password);
            }

            this.m_error = welcome;
            return -1;
        }

        #endregion
    }
}

The methods of class POP3Client were not virtual, but fortunately some of the class members were in protected scope and were, therefore, accessible in the new class. I rewrote the Open and Quit methods as new methods. This made them no longer polymorphic, thus forcing us to replace calls to POP3Client with calls to Pop3ClientWithSsl everywhere in the code.

Java Server Faces rage!

From http://thoughtworks.fileburst.com/assets/technology-radar-jan-2014-en.pdf:

We continue to see teams run into trouble using JSF– JavaServer Faces — and are recommending you avoid this technology.

Teams seem to choose JSF because it is a J2EE standard without really evaluating whether the programming model suits them. We think JSF is flawed because it tries to abstract away HTML, CSS and HTTP, exactly the reverse of what modern web frameworks do. JSF, like ASP.NET webforms, attempts to create statefulness on top of the stateless protocol HTTP and ends up causing a whole host of problems involving shared server-side state. We are aware of the improvements in JSF 2.0, but think the model is fundamentally broken.
We recommend teams use simple frameworks and embrace and understand web technologies including HTTP, HTML and CSS.

This is exactly how I am feeling about Java Server Faces.

You can see from my old posts that I have been a strong supporter of JSF from the beginning. I continued to believe in its potential even when it was new and lacked features.

JSF 2.0 was supposed to address the shortcomings with, for example, bookmarkable URLs and view parameters. But JSF 2.0 still has many flaws, and I still have to jump through hoops and deal with incompatible implementations despite the specifications of JavaEE to get certain things to work.

Model-Based Testing

Robert Binder’s Testing Object-Oriented Systems book sits permanenly on my desk. At over 1500 pages long, it is almost a never-ending read, but from time to time I pause to read a few choice chapters.

Binder also wrote about the compliance testing of Microsoft’s court-ordered publication of its Windows client-server protocols in 2012. An interesting fact from the article is that instead of having to test software against documentation, they had to do the reverse because the code was already published and had to be used as the gold standard. Under scrutiny and tight deadlines, they managed to check that 60,000 pages of documentation matched the protocols exactly, all thanks to model-based testing (MBT).

Test fixtures

I try my best to avoid the Arrange-Act-Assert (AAA) pattern for unit tests. I find that with multiple test methods needing the same starting conditions, the ‘arrange’ code must be repeated too much. This makes writing code tedious and complicates later changes.

I prefer to initialise a test fixture to a common initial state that each test method can start from. Like the real thing, a test fixture keeps the state steady as the tests are executed.

For example, a fixture to test ‘new customer’ scenarios will be done as class NewCustomerTests, and a fixture to test ‘existing customer’ scenarios as class ExistingCustomerTests.

Most test frameworks allow a method in a test class to be run before each test method is executed. In JUnit, the decorator @Before  identifies this method; in MSTest, the attribute [TestInitialize] does the same.

I believe that using test fixtures to organise tests around common initial conditions makes test code more organised and maintainable.

The unit in unit-testing

The interpretation of “unit” in unit testing to mean a class is wrong. This and an unfortunate misunderstanding of Kent Beck’s “run unit tests in isolation” still drive developers to test their classes in extreme isolation with undesirable consequences that are then attributed to test-driven development (TDD) as being unsustainable.

When we make our test units too small, we have to rely on mocks and stubs to stand for collaborators that are needed by the classes being tested. As the size of our test suite grows, so do the number and usage of mocks and stubs. There are two major problems with those.

First, when we use them in tests, we reveal the implementation details of the classes that depend on them. This is because we write expectations and verifications based on a very detailed knowledge of how the mocks and stubs are used.

Second, unlike real objects that change organically in response to changes in their collaborations, mocks and stubs are resistant to change due to the very specific ways that they are set up for test cases.

The first problem is a compromise that developers usually accept in exchange for testability. Also, many consider it to be harmless because encapsulation is still preserved.

On the other hand, the second problem is what hurts developers most and drives many to abandon TDD. The scale of the problem manifests itself once the test suite has reached a considerable size. At that point, any slight change in the nature of the collaboration between classes requires changes to numerous tests that use mocks and stubs to fake the collaboration, with many expectations and verifications suddenly needing revision. Since the task is usually tedious, and mocks and stubs do not have actual value in production, there is little motivation to spend time on maintaining the tests. Eventually the practice of TDD within the team disappears altogether.

Ideally, mocks and stubs would only be used for testing integration between components instead of collaboration between classes, but if their use cannot be avoided, their effect can at least be minimised. The most obvious way to do this is to lower the number of expectations that are set up in the test suite. By grouping tests according to the mocks and stubs that they use and the expectations that are needed, respectively, the number of test cases that need to be changed when collaborations change will be less.

There are other ways to make TDD more efficient, such as good discipline in the design of test cases, but one improvement that can be applied immediately is to reduce the use of mocks and stubs. And the way to do that is to exercise collaboration of actual classes in tests.

Learning BASE64 encoding

BASE64 encoding is so prevalent that it is worth learning how it works and how to code your own implementation.

The point of BASE64 is to communicate binary data as text, using only characters that are likely to exist on most computer platforms. These safe characters are known as the BASE64 alphabet and are the letters A to Z and a to z, the numerals 0 to 9, and the characters / and +. There are other ways to represent bytes as text; for example, by converting them to hexadecimal strings made up of the characters 0 to 9 and A to F. But doing so means that for every character in the original set, two hexadecimal characters are required, which doubles the size of the data.

The BASE64 alphabet consists of 64 characters, each one associated with an integer value. For example, the character A is represented by 0, the character Z by 25, and character / by 63. This means that to cover the range of integers from 0 to 63, the BASE64 word size must be six bits (because 2^6=64). As a consequence of this, during BASE64 encoding the original data must be laid out and padded to make its size — in bits — a number divisible by six.

The smallest number of bytes (or 8-bit words) that can be re-arranged in groups of 6-bit words is three (3×8 bits = 24 bits, which is divisible by six). In other words, data must be processed in groups of 24 bits, each group being equivalent to four 6-bit words (4×6 bits = 24 bits). The BASE64 character matching the value of each 6-bit word is then output as an 8-bit ASCII character. So for every three bytes of input, four bytes of output are generated, giving an inflation factor of 4:3 (which is a better compromise than the 2:1 ratio from hexadecimal encoding).

Data that cannot be split exactly in groups of 24 bits must be padded to make them so. For example, data that are one byte long (i.e. 8 bits) must be padded with two zero-value bytes (i.e. 8 bits + (2×8 bits)), and data that are 11 bytes long (i.e. 88 bits) must be padded with one zero-value byte (i.e. 88 bits + 8 bit = 96 bits = 4×24 bits). In other words, data must be padded to reach a size that is divisible by three.

With the theory out of the way, here is how BASE64 is implemented in Java, using the example ‘any carnal pleasure’.

First, encode the string as a series of bytes.

byte[] bytes = "any carnal pleasure".getBytes();

This results in an array of 19 bytes.

Next, pad the array with two zero-value bytes to make its size divisible by three.

byte[] padded = Arrays.copyOf(bytes, 21);

Then, convert each triplet of bytes into four 6-bit words and calculate the value of each. (Use bit shift operators.) Append the BASE64 character represented by each 6-bit value to a StringBuilder instance.

for (int byteIndex = 0; byteIndex < padded.length; byteIndex += byteGroupSize) {

    // read the value of the 24-bit word starting at the current index
    int wordOf24Bits = (padded[byteIndex] << 16) + 
                       (padded[byteIndex + 1] << 8) + 
                        padded[byteIndex + 2];

    // read the 24-bit word as 6-bit word values
    int wordOf6Bits1 = (wordOf24Bits >> 18) & 63;
    int wordOf6Bits2 = (wordOf24Bits >> 12) & 63;
    int wordOf6Bits3 = (wordOf24Bits >>  6) & 63;
    int wordOf6Bits4 = (wordOf24Bits      ) & 63;

    result.append(BASE64_CHARS.charAt(wordOf6Bits1));
    result.append(BASE64_CHARS.charAt(wordOf6Bits2));
    result.append(BASE64_CHARS.charAt(wordOf6Bits3));
    result.append(BASE64_CHARS.charAt(wordOf6Bits4));
}

This yields the BASE64 string ‘YW55IGNhcm5hbCBwbGVhc3VyZQAA’.

Finally, replace the padding characters (“AA” in this example resulting from the two zero-value bytes) with as many “=” characters. The “=” is used in the BASE64 decoding process (which is not covered in this post) to determine the amount of padding that has been applied.

for (int i = result.length(); i > result.length() - paddingSize; i--) {
    result.setCharAt(i - 1, '=');
}

This gives the final result ‘YW55IGNhcm5hbCBwbGVhc3VyZQ==’.

I know that there are at least two classes in the standard Java libraries that provide BASE64 operations. One of those is undocumented and is subject to change, and the other is meant to be used by the mail library, which could cause confusion (or would be bad form?) if they are referenced in code that does not otherwise depend on the libraries where the classes reside. By writing my own implementation, I can avoid these unnecessary dependencies, and most importantly, I can do BASE64 in any language that does not have a built-in function for it.

How we use SQL Server Data Tools

This post describes the process that we use to develop databases with SQL Server Data Tools (SSDT) in Visual Studio.

For this process to work, the conventions below must be respected.

  • Use the live database as the gold standard for schema objects (and data).
  • Deploy only database projects that have been built successfully.
  • Deploy to a database that matches the schema of the live database.

At the beginning of a development iteration

  1. Restore a copy of the live database onto the development computer.
  2. Synchronise database project schema objects with the schema objects in the restored database.
  3. Remove pre-deployment and post-deployment scripts from the database project.
  4. Update the database project version number.
  5. Build the database project.
  6. If the build fails, fix the errors and rebuild.
  7. If the build completes, check in the changes.

During a development iteration

  1. Make changes to script files in the database project.
  2. If the changes might result in data loss, write pre-deployment and post-deployment scripts to migrate the data.
  3. Build the database project.
  4. If the build fails, fix the errors and rebuild.
  5. If the build succeeds, publish the changes onto the database on the development computer and test.

Interim releases to the test environment

  1. Restore a copy of the live database from backup.
  2. Build the database project.
  3. Publish the database project onto the test server.

Deployment to the live environment

  1. Back up the live database.
  2. Build the database project.
  3. Publish the database onto the live server.