Restarting Bluetooth on Debian Linux

On my laptop I use a Bluetooth keyboard and a Bluetooth mouse. Occasionally the Bluetooth adapter in the laptop shuts off completely, which means that it is not even possible to restart the Bluetooth service with sudo systemctl restart bluetooth.service.

My troubleshooting points to powersaving as the cause of this, but the TLP setting that is supposed to prevent autosuspend on Bluetooth devices does not seem to prevent this sudden death of the Bluetooth connection. As a temporary workaround, I am using the commands below to restart Bluetooth from the shell.

$ sudo rmmod btusb && sudo modprobe btusb && sudo systemctl restart bluetooth.service

This solution works on Debian testing bullseye with kernel x86_64 Linux 5.5.0-2-amd64.

Centrino Ultimate N-6300 slow to connect to Wi-Fi network

Wi-Fi connections from my laptop became unstable after I upgraded it to Ubuntu 19.04. To get them to work, I had to turn the Wi-Fi adapter on and off several times, which quickly became annoying.

The following attempts to fix the problem were unsuccessful:

  • disabling IPv6
  • disabling power-saving on the Wi-Fi adapter
  • setting bt_coex_active=0 during kernel loading to disable Bluetooth/Wi-Fi interference protection

Eventually the correct solution was to disable 802.11n with the following procedure:

  1. Edit or create a configuration file under /etc/modprobe.d (e.g. iwlwifi_custom.conf).
  2. Add this line and save the file.
    options iwlwifi 11n_disable=1
  3. Reload the iwlwifi module.
    sudo rmmod iwldvm
    sudo rmmod iwlwifi
    sudo modprobe iwlwifi
  4. Connect to the Wi-Fi network.

SO #57225262—Inputting list of passwords in a class

Stack Overflow question ‘Inputting list of passwords in a class’ is more interesting as an object-oriented design (OOD) exercise than as a debugging exercise, but it also provides an opportunity to practise some Python programming.

The requirements identified in the problem are:

  • represent a set of credentials comprised of a username and a password;
  • represent a user account made up of an identifier and one active set of credentials and zero or more inactive credentials;
  • store user accounts that can be retrieved subsequently by username;
  • validate a given set of credentials, using the stored user accounts.

Additionally, a user account has the following constraints:

  • It must be uniquely identifiable by the username of the active set of credentials.
  • A set of credentials must be used only once for a given user account.

The design model consists of multiple classes, each addressing a single responsibility as follows.

Class Responsibility
Credentials represent a set of credentials
UserAccount represent a user account
AccountStore store user accounts, enforcing the uniqueness constraint
Authentication validate supplied credentials

The program code implements the model very faithfully and uses the same class names, thus making it easy to reason about. It has been tested with Python 2.7.15rc1 and has been checked for PEP8 compliance.

NOTES

For brevity, the solution presented here does not:

  • implement best practices used for security in real-world scenarios, such as salting and hashing;
  • enforce uniqueness of user account identifiers;
  • consider performance factors.

In lieu of conventional unit tests, a main routine exercises use-cases to ensure that the program works properly.

class Credentials:

    def __init__(self, username, password):
        self.username = username
        self.password = password

    def has_username(self, username):
        return self.username == username

    def matches(self, credentials):
        return self.username == credentials.username and \
            self.password == credentials.password

class UserAccount:

    def __init__(self, user_id):
        self.user_id = user_id
        self.active_credentials = None
        self.past_credentials = []

    def add(self, credentials):
        self._check_uniqueness(credentials)
        if self.active_credentials is None:
            self.active_credentials = credentials
        else:
            self.past_credentials.append(self.active_credentials)
            self.active_credentials = credentials

    def has_username(self, username):
        return self.active_credentials.has_username(username)

    def has_same_username(self, user_account):
        return self.has_username(user_account.active_credentials.username)

    def has_credentials(self, credentials):
        return self.active_credentials is not None and \
            self.active_credentials.matches(credentials)

    def _check_uniqueness(self, credentials):
        if self.has_credentials(credentials):
            raise Exception('These credentials are currently in use.')
        for c in self.past_credentials:
            if c.matches(credentials):
                raise Exception(
                        'These credentials have been used in the past.')

class AccountStore:

    def __init__(self):
        self.user_accounts = []

    def add(self, user_account):
        self._check_uniqueness(user_account)
        self.user_accounts.append(user_account)

    def find_by_username(self, username):
        for ua in self.user_accounts:
            if ua.has_username(username):
                return ua
        return None

    def _check_uniqueness(self, user_account):
        for ua in self.user_accounts:
            if ua.has_same_username(user_account):
                raise Exception(
                        'An account with the same username is already active.')

class Authentication:

    def __init__(self, account_store):
        self.account_store = account_store

    def validate(self, credentials):
        user_account = self.account_store.find_by_username(
                credentials.username)
        if user_account is None:
            return False
        return user_account.has_credentials(credentials)

if __name__ == '__main__':
    credentials = Credentials('user1', 'password1')
    user_account = UserAccount(101)
    user_account.add(credentials)

    account_store = AccountStore()
    account_store.add(user_account)

    user_account1 = account_store.find_by_username('user1')
    print 'user_account1', user_account1

    user_account2 = account_store.find_by_username('user2')
    print 'user_account2', user_account2

    authentication = Authentication(account_store)
    print 'Expecting True...', authentication.validate(
            Credentials('user1', 'password1'))
    print 'Expecting False...', authentication.validate(
            Credentials('user2', 'password1'))
    print 'Expecting False...', authentication.validate(
            Credentials('user1', 'password2'))

    user_account.add(Credentials('user1', 'password2'))
    print 'Expecting True...', authentication.validate(
            Credentials('user1', 'password2'))
    print 'Expecting False...', authentication.validate(
            Credentials('user1', 'password1'))

    try:
        user_account.add(Credentials('user1', 'password1'))
    except Exception:
        print 'Expecting exception... Pass'

    try:
        user_account.add(Credentials('user2', 'password1'))
        print 'Not expecting exception... Pass'
        print 'Expecting True...', authentication.validate(
                Credentials('user2', 'password1'))
    except Exception:
        print 'Not expecting exception... Fail'

    try:
        user_account1 = UserAccount(102)
        user_account1.add(Credentials('user1', 'whatever'))
        account_store.add(user_account1)
        print 'Expecting True...', authentication.validate(
                Credentials('user1', 'whatever'))
    except Exception:
        print 'Not expecting exception... Fail'

    try:
        user_account2 = UserAccount(103)
        user_account2.add(Credentials('user1', 'whatever'))
        account_store.add(user_account1)
        print 'Expecting exception... Fail'
    except Exception:
        print 'Expecting exception... Pass'


The output of the program is:

EY@LENNY:~/Source/junk/python/pwman$ python all.py
user_account1 <__main__.UserAccount instance at 0x7faa36f11170>
user_account2 None
Expecting True... True
Expecting False... False
Expecting False... False
Expecting True... True
Expecting False... False
Expecting exception... Pass
Not expecting exception... Pass
Expecting True... True
Expecting True... True
Expecting exception... Pass
EY@LENNY:~/Source/junk/python/pwman$

Programming Arduino in Atmel Studio

As a fun way to improve my C, I started programming the Arduino using Atmel Studio instead of the friendlier Arduino IDE.

Below is my very first program. It blinks a red LED and a white LED according to a preset pattern.

#define F_CPU 16000000UL

#include <avr/io.h>
#include <util/delay.h>

struct Led
{
    int pin;
    int blinks;
    int on_duration;
    int off_duration;
};

void delay_ms(int ms)
{
    while (ms-- > 0) {
        _delay_ms(1);
    }
}

void blink(const struct Led* const led)
{
    for (int i = 0; i < led->blinks; i++) {
        PORTB |= led->pin;
        delay_ms(led->on_duration);
        PORTB &= ~led->pin;
        delay_ms(led->off_duration);
    }
}

int main(void)
{
    const struct Led white_led = { 1<<PB6, 10, 100, 500 };
    const struct Led red_led = { 1<<PB7, 10, 500, 1000 };

    DDRB |= white_led.pin;
    DDRB |= red_led.pin;
    while (1) {
        blink(&white_led);
        blink(&red_led);
    }
}

With C in Atmel Studio and the AVR-LIBC library, IO ports are manipulated by changing bit patterns in the registers associated with the relevant ports. This requires a good understanding of bitwise operations in C despite the availability of macros to simplify the task.

For example, to set pin 12 of the Arduino to output mode, bit 6 of register DDRB must be set to 1. To do so requires an | (OR) operation with a bit pattern operand where bit 6 is set to 1 and the rest set to 0, so that the states of the other bits in the register are not disturbed.

Using macros DDRB and PB6 defined in AVR-LIBC, this is done like this: DDRB |= 1 << PB6 .

If you are new to C and are unfamiliar with macros, you might wonder about that statement. Besides, DDRB and PB6 are not referenced anywhere else in my program, so how does this line of code work?

DDRB is a macro that expands into C code to dereference the address of the register associated with setting the mode for pin 12, and PB6 is just a symbolic constant for the value 6. In the statement above, by shifting the value 1 left by 6 positions, we create a new value which is then applied to the bit pattern stored at the dereferenced address with an | operation to turn bit 6 of the register to 1. In this case, this sets pin 12 to output mode.

In a nutshell, the sequence of operations is as follows.

Step 1:

1 << 6 = 01000000

Step 2:

Assuming the register DDRB is initially 00000001:

00000001 | 01000000 = 01000001

In my C program, the result of step 1 is assigned to struct field Led.pin and is used as the second operand for the operation in step 2.

It took about an hour to refresh my knowledge of bitwise operations, but the real challenge was interpreting the Arduino schema and the information in datasheets, especially to find the right registers to manipulate.

 

 

Hacking SSL support into smtpop.dll

We use smtpop.dll in one of our applications to retrieve email from POP3 mailboxes. Today we had to connect to a mailbox over SSL, which smtpop.dll does not support.

The code to retrieve email sits behind a façade, so I expected that it would be simple to replace smtpop.dll with another library and call the new classes. But, because the façade was  tightly coupled with the interface of smtpop.dll, we needed a replacement with the same interface.

After further tests, I decided to decompile smtpop.dll, examine how it worked internally, and derive a new class from it. It turned out that two methods, Open and Quit, would need to be changed.

namespace ClassLibrary4
{
    using System.IO;
    using System.Net.Security;
    using System.Net.Sockets;

    using SmtPop;

    public class Pop3ClientWithSsl : POP3Client
    {
        #region Fields

        private SslStream sslStream;

        #endregion

        #region Constructors and Destructors

        public Pop3ClientWithSsl()
        {
            this.UseSsl = true;
        }

        #endregion

        #region Public Properties

        public bool UseSsl { get; set; }

        #endregion

        #region Public Methods and Operators

        public new int Open(string hostname, int port, string username, string password)
        {
            if (this.UseSsl)
            {
                return this.OpenWithSsl(hostname, port, username, password);
            }

            return base.Open(hostname, port, username, password);
        }

        public new string Quit()
        {
            try
            {
                return base.Quit();
            }
            finally
            {
                this.m_streamReader.Close();
                this.m_streamWriter.Close();
                if (this.UseSsl)
                {
                    this.sslStream.Close();
                }
            }
        }

        #endregion

        #region Methods

        private int OpenWithSsl(string hostname, int port, string username, string password)
        {
            this.m_host = hostname;
            this.m_port = port;
            this.m_user = username;
            this.m_tcpClient = new TcpClient(hostname, port);

            this.m_netStream = this.m_tcpClient.GetStream();
            this.sslStream = new SslStream(this.m_netStream, false);
            this.sslStream.AuthenticateAsClient(hostname);

            this.m_streamReader = new StreamReader(this.sslStream);
            this.m_streamWriter = new StreamWriter(this.sslStream) { AutoFlush = true };

            string welcome = this.m_streamReader.ReadLine();
            if (welcome != null && welcome.StartsWith(+OK))
            {
                return this.SendLogin(username, password);
            }

            this.m_error = welcome;
            return -1;
        }

        #endregion
    }
}

Unfortunately, the methods of class POP3Client are not virtual. But, luckily for us, some of the class members are of protected visibility and so accessible in the derived class. I rewrote the Open and Quit methods as new methods, which means that they are not polymorphic. This also forces us to replace usage of POP3Client with Pop3ClientWithSsl everywhere in the code. But, at least, we have some respite before we have to implement a cleaner solution.

Learning BASE64 encoding

BASE64 encoding is so prevalent that it is worth learning how it works and how to code your own implementation.

The point of BASE64 is to communicate binary data as text, using only characters that are likely to exist on most computer platforms. These safe characters are known as the BASE64 alphabet and are the letters A to Z and a to z, the numerals 0 to 9, and the characters / and +. There are other ways to represent bytes as text; for example, by converting them to hexadecimal strings made up of the characters 0 to 9 and A to F. But doing so means that for every character in the original set, two hexadecimal characters are required, which doubles the size of the data.

The BASE64 alphabet consists of 64 characters, each one associated with an integer value. For example, the character A is represented by 0, the character Z by 25, and character / by 63. This means that to cover the range of integers from 0 to 63, the BASE64 word size must be six bits (because 2^6=64). As a consequence of this, during BASE64 encoding the original data must be laid out and padded to make its size — in bits — a number divisible by six.

The smallest number of bytes (or 8-bit words) that can be re-arranged in groups of 6-bit words is three (3×8 bits = 24 bits, which is divisible by six). In other words, data must be processed in groups of 24 bits, each group being equivalent to four 6-bit words (4×6 bits = 24 bits). The BASE64 character matching the value of each 6-bit word is then output as an 8-bit ASCII character. So for every three bytes of input, four bytes of output are generated, giving an inflation factor of 4:3 (which is a better compromise than the 2:1 ratio from hexadecimal encoding).

Data that cannot be split exactly in groups of 24 bits must be padded to make them so. For example, data that are one byte long (i.e. 8 bits) must be padded with two zero-value bytes (i.e. 8 bits + (2×8 bits)), and data that are 11 bytes long (i.e. 88 bits) must be padded with one zero-value byte (i.e. 88 bits + 8 bit = 96 bits = 4×24 bits). In other words, data must be padded to reach a size that is divisible by three.

With the theory out of the way, here is how BASE64 is implemented in Java, using the example ‘any carnal pleasure’.

First, encode the string as a series of bytes.

byte[] bytes = "any carnal pleasure".getBytes();

This results in an array of 19 bytes.

Next, pad the array with two zero-value bytes to make its size divisible by three.

byte[] padded = Arrays.copyOf(bytes, 21);

Then, convert each triplet of bytes into four 6-bit words and calculate the value of each. (Use bit shift operators.) Append the BASE64 character represented by each 6-bit value to a StringBuilder instance.

for (int byteIndex = 0; byteIndex < padded.length; byteIndex += byteGroupSize) {

    // read the value of the 24-bit word starting at the current index
    int wordOf24Bits = (padded[byteIndex] << 16) + 
                       (padded[byteIndex + 1] << 8) + 
                        padded[byteIndex + 2];

    // read the 24-bit word as 6-bit word values
    int wordOf6Bits1 = (wordOf24Bits >> 18) & 63;
    int wordOf6Bits2 = (wordOf24Bits >> 12) & 63;
    int wordOf6Bits3 = (wordOf24Bits >>  6) & 63;
    int wordOf6Bits4 = (wordOf24Bits      ) & 63;

    result.append(BASE64_CHARS.charAt(wordOf6Bits1));
    result.append(BASE64_CHARS.charAt(wordOf6Bits2));
    result.append(BASE64_CHARS.charAt(wordOf6Bits3));
    result.append(BASE64_CHARS.charAt(wordOf6Bits4));
}

This yields the BASE64 string ‘YW55IGNhcm5hbCBwbGVhc3VyZQAA’.

Finally, replace the padding characters (“AA” in this example resulting from the two zero-value bytes) with as many “=” characters. The “=” is used in the BASE64 decoding process (which is not covered in this post) to determine the amount of padding that has been applied.

for (int i = result.length(); i > result.length() - paddingSize; i--) {
    result.setCharAt(i - 1, '=');
}

This gives the final result ‘YW55IGNhcm5hbCBwbGVhc3VyZQ==’.

I know that there are at least two classes in the standard Java libraries that provide BASE64 operations. One of those is undocumented and is subject to change, and the other is meant to be used by the mail library, which could cause confusion (or would be bad form?) if they are referenced in code that does not otherwise depend on the libraries where the classes reside. By writing my own implementation, I can avoid these unnecessary dependencies, and most importantly, I can do BASE64 in any language that does not have a built-in function for it.

How we use SQL Server Data Tools

This post describes the process that we use to develop databases with SQL Server Data Tools (SSDT) in Visual Studio.

Conventions

For this process to work, the conventions below must be followed.

  • Use the live database as the gold standard for schema objects (and data).
  • Deploy only database projects that have been built successfully.
  • Deploy to a database that matches the schema of the live database.

At the beginning of a development iteration

  1. Restore a copy of the live database onto the development computer.
  2. Synchronise database project schema objects with the schema objects in the restored database.
  3. Remove pre-deployment and post-deployment scripts from the database project.
  4. Update the database project version number.
  5. Build the database project.
  6. If the build fails, fix the errors and rebuild.
  7. If the build completes, check in the changes.

During a development iteration

  1. Make changes to script files in the database project.
  2. If the changes might result in data loss, write pre-deployment and post-deployment scripts to migrate the data.
  3. Build the database project.
  4. If the build fails, fix the errors and rebuild.
  5. If the build succeeds, publish the changes onto the database on the development computer and test.

Interim releases to the test environment

  1. Restore a copy of the live database from backup.
  2. Build the database project.
  3. Publish the database project onto the test server.

Deployment to the live environment

  1. Back up the live database.
  2. Build the database project.
  3. Publish the database onto the live server.

How to identify and fix an anaemic domain model

CRUD applications tend to involve classes with many accessor methods but little business logic, the behaviour being implemented at a higher level in manager (or ‘god’) classes. Usually this signals that the domain model is anaemic. While many developers know the symptoms, few know how to resolve the problem.

Anaemic classes have no responsibility other than to write and read attributes. Not only the utility of object-oriented programming is diminished in such cases, but the amount of boilerplate code also increases to implement data transfer objects and manager classes to move them around.

Unit-testing can help to identify when a domain model is anaemic.

Unit testing demands that the ‘usage code’ be written before the classes are implemented. By first writing tests, one can see how the interfaces of a domain model will be used and can spot easily anything that is out of place or redundant. For example, consider a simple Account class that has the following responsibilities:

  • represent a user account
  • hold information about a user (ie. username, password, email address, status)
  • used for authentication

This class seems dumb at first, and it is tempting to implement only getter and setter methods as the interface.

Say, we write a unit test for user authentication. What most people probably do is this:

assertTrue(account.getPassword().equals("testpassword"));

This, in the unit test, should immediately signal that the model is anaemic. Here the responsibility for checking the validity of the input password is handled by the calling code, but in truth it should belong to the Account class.

An easy fix is to refactor the unit test as follows.

assertTrue(account.hasPassword("testpassword"));

Now the behaviour is encapsulated within the Account class, making the implementation of the ‘authenticate user’ use-case more robust.

Many such small refactorings can be performed in order to make richer classes. One has only to write good unit tests and refactor whenever one thinks that things can be written in a better way.

I follow these principles:

  • If calling code reads an attribute from an object in order to act on that piece of information, the behaviour should be moved to the object.
  • Getters and setters should not be systematically implemented; instead, write setters and setters when unit tests indicate that they are needed.
  • If methods on the same object are called in a daisy-chain fashion, this indicates that the object is anamic and lacks behaviour.

I am sure other programmers have their own ways of resolving anaemic domain models. I would be very interested in reading what techniques they use.