Case Genie or How to Find Unknown Unknowns

It was Paul Magrath, Head of Product Development and Online Content at ICLR, who first used the late Donald Rumsfeld’s phrase to describe the business case for what later became known as Case Genie.

The idea was simple. Lawyers needed to discover historical cases that might impact a case they were preparing. Frequently-cited cases would already be known to them: they’d be at the tips of their fingers, ready to type into their skeleton argument; cases known to every barrister specialising in the field they were arguing; cases that had been cited many times both by other cases and in text books; or cases that had recently changed how the law should be interpreted, and were therefore big news within a narrowly-focused legal community.

But there may be some cases that are relatively unknown that might make all the difference. Enter Case Genie.

This blog post presents a technical overview of Case Genie. How, exactly, is it possible to find “unknown unknowns”?

Word embeddings

Document embeddings are considered the state-of-the-art way of finding similarities between documents. But before document embeddings come word embeddings. A good way to start to think about word embeddings, is to think about words in a document. Let’s take Milton’s Areopagitica as an example. This single work is our corpus (the body of text we’re interested in). Let’s take a single sentence from Milton’s Areopagitica:

Many a man lives a burden to the Earth; but a good Book is the precious life-blood of a master spirit, embalmed and treasured up on purpose to a life beyond life.

Now, if we take each distinct word, we can give it a score based on how often it occurs in the text:

  many  — 1
  a     — 5
  man   — 1
  lives — 1
  …etc…

Now imagine graphing three of these words. I’m choosing three because this is easy to visualise, but rather than take the first three words in the sentence, let’s take the following:

  a         — 5
  life      — 3
  treasured — 1

Imagine these graphed across the x, y and z axes: ‘a’ has the value 5 on the x axis; ‘life’ has the value 3 on the y axis; and ‘treasured’ has the value 1 on the z axis. Further, imagine that for each value, there is an arrow from the origin to the point along each axis, so that we have 3 arrows of different lengths pointing in different directions. Now, in mathematics a value with a direction is called a vector, so we can say that each distinct word within a corpus can be represented as a vector; and within a document, a vector’s value is the number of occurrences of the word within that document.

Three dimensions aren’t too hard to visualise. Extrapolating, we can add further axes, which is much harder to visualise; as many axes as there are distinct words. Each distinct word in the corpus will therefore be represented by a vector.

Of course, that’s just one sentence and there are many in Areopagitica. The full vector space is defined by the number of unique words in the corpus, let’s suppose there are 2,000 in Areopagitica. Initially this will sound confusing, but the word embedding for a given word is made up of a value for every vector in the vector space, which is the same as saying a value for every word. For each distinct word, its word embedding will therefore comprise 1,999 zero values and one non-zero value — 1. If we order the distinct words alphabetically, we can represent an embedding as an implicitly ordered array of numbers. Therefore, for each word embedding, the non-zero value will be in a different place; ‘a’ would be:

  1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 …

whereas ‘and’ would be:

  0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 …

This is crazily sparse; the cleverness comes when using AI to use all that empty space more wisely.

Algorithms like word2vec and fastText (there are other algorithms, but these are the two we looked at for ICLR) provide a more compact representation of word embeddings. Whereas with Areopagitica, using the algorithm I described above, we would need 2,000 values (dimensions) for each word, word2vec and fastText can use a few hundred dimensions. You can download a 300-dimensional model from the fastText website trained on 2 million words from Wikipedia. Word embeddings actually use floating point numbers, so the word ‘a’ in the fastText model we trained for ICLR starts:

  -0.20005 -0.019533 -0.15494 -0.11114 0.074793 0.1194 -0.046101 …

The model for ICLR has 600 dimensions, so there are 600 numbers for each word embedding.

Exactly how word2vec and fastText create these word vectors is outside the scope of this blog post. Essentially, you start by training a model from the corpus, which you feed to the tool. The corpus and the training algorithms define relationships between words or character combinations. The corpus used to train the model therefore determines how the words’ relationships are actually represented in the model. If you train with a French corpus, you will get good results when calculating document embeddings for French text; but you would get bad results if you presented English text. In the same vein, if you train a model using legal documents, you will get a model that is more sensitive to legal meanings and definitions.

So, if we were to use Areopagitica as our corpus, we would have a model trained to recognise only uses of the words as used in Areopagitica. We could calculate sentence embeddings (which we could treat like document embeddings) for every sentence in Areopagitica to determine which sentences were most similar. However, the results would likely be quite poor. The reason for this is that the corpus is too small. Ideally, you need a lot of words, the more the better. For ICLR, we used all of the judgment transcripts and Case Reports contained in ICLR.online as our corpus. This represents around 200,000 documents; about 600,000,000 words.

But what are document embeddings, and how do you create them?

Document embeddings

Once you have word embeddings, you will want to create document embeddings. To create a document embedding, take all the word embeddings of the document and use cosine similarity to combine the dimensions into a single representation with the same number of dimensions.

Cosine similarity combines two vectors by calculating the cosine of the angle between them and multiplying by the length of the side not participating in the cosine calculation.

Cosine similarity is a nice way to explain it visually, but there is an algebraic isomorphism using dot products of normalised vectors.

This algorithm combines word embeddings into document embeddings:

  1. Create an empty embedding (where all values are 0) called A. This is our aggregate.
  2. For each word embedding, w:
    1. normalise w;
    2. for each value in A, add the corresponding value in w (add the first number in A to the first number in w, the second number in A to the second number in w, and so on).
  3. Divide each value in A by the number of words used to calculate it.

To normalise a word embedding, take the square root of the sum of each value’s square (let’s call it n); then divide each value in the embedding by n.

This is the implementation used by fastText. However we have tweaked the algorithm to make use of tf–idf. tf–idf weights words that are considered important. A word is considered important if its frequency within the corpus is low compared to its frequency within a document. Since words like ‘a’ occur throughout the corpus, it is weighted low; whereas a word like ‘theft’ will be weighted higher, because it occurs only in a subset of documents within the corpus.

Calculating the distance between document embeddings

The final puzzle piece is to quantify how similar documents are based on their document embeddings. The distance between two documents is used as an indication of how similar they are. Documents that are close together are more similar than those that are further away.

To calculate the distance between two document embeddings, Cosine Similarity can be used again. This time, the vectors of the document embeddings are collapsed to a single value between 0 and 1. Identical documents have the value 1, completely orthogonal documents have the value 0.

We use a library called Faiss, created by Facebook Research, to store and query document embeddings. This is very fast, and you can query multiple input embeddings simultaneously.

And finally…

This post describes some of the major components used in Case Genie and delves a little into the concepts and some of the algorithms; but there is a lot more that I don’t have time to cover: specifically, how do we prepare the text of the corpus for training the model? I will cover that in another post.

Understanding an “impossible” error

As discussed in the previous post on Sharing Failures, seeing how other people have dealt with bugs and errors can often help you avoid your own or give you ways to track down the source of a problem when one does make its appearance. So in that spirit, here is the story of a baffling error we fixed recently.

The error came from a content delivery platform we have been working on for a publisher client. At the point of a release and for several hours after we were seeing some errors, but there were a few reasons why this was very confusing.

The site is built using Scala / Play and uses Akka HTTP to make API calls between services. The error we were seeing was one that generally means that requests are coming in to a frontend service faster than the backend can service them:

BufferOverflowException: Exceeded configured max-open-requests value of [256]. This means that the request queue of this pool (........) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See https://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.]]

So apparently the pool of requests was filling up and causing a problem. But the first thing that was strange was that this was persisting for several hours after the release. At the point of a release it’s understandable that this error could occur with various services being started and stopped, causing requests to back up. After that the system was not under particularly high load, so why was this not just a transient issue?

The next thing that was strange was that we were only seeing this when users were accessing very particular content. We were only seeing it for access to content in reference works. These are what publishers confusingly call “databases” and cover things like encyclopedias, directories or dictionaries. But it wasn’t all databases, only certain ones and different ones at different times. On one occasion we would see a stream of errors for Encyclopedia A and then the next time we hit this error it would be Dictionary B generating the problems instead. If the cause was a pool of requests filling up, why would it affect particular pieces of content and not others, when they all use the same APIs?

Another thing that was puzzling – not every access to that database would generate an error. We’d either get an error or the content would be rendered fine, both very quickly. The error we were seeing suggested that things were running slowly somewhere, but the site seemed to be snappy, just producing intermittent errors for some content some of the time.

We spent lots of time reading Akka HTTP documentation trying to figure out how we could be seeing these problems, but it didn’t seem to make any sense. I had the feeling that I was missing something because the error seemed to be “impossible”. I even commented to a colleague that it felt like once we worked out what was going on I would talk about it at one of our dev forums. That prediction turned out to be true. Looking at Akka HTTP documentation would not help because the error message itself was in some sense a misdirection.

The lightbulb moment came when I spotted this code in our frontend code:

private lazy val databaseNameCache: LoadingCache[String, Future[DatabaseIdAndName]] = 
    CacheBuilder.newBuilder().refreshAfterWrite(4, TimeUnit.HOURS).....

We are using Guava’s LoadingCache to cache the mapping between the id of a database and its name since this almost never changes. (Sidenote: Guava’s cache support is great, also check out the Caffeine library inspired by it). The problem here is that we are not storing a DatabaseIdAndName object in the cache, but a Future. So we are in some sense putting the operation to fetch the database name into the cache. If that fails with an Exception, then every time we look in the cache for it we will replay the exception. Suddenly all the pieces fell into place. A transient error looking up a database name at release time was being put in a cache on one frontend server and replayed for hours. The whole akka pool thing was more or less irrelevant.

In the short term we fixed the problem by waiting for the concrete data to be returned to store that in the cache rather than a Future object. In that scenario, a failure to fetch the value would just yield an error and nothing would be cached for future look ups. However, much of the code using this cache is asynchronous, so it’s cleaner and probably better from a performance perspective if you can continue to use Future where possible. So the longer term solution was to revert to putting Future objects in the cache but carefully adding code to invalidate any cache entries that resolve to an exception.

I think the lesson here is – if an error doesn’t make sense then maybe some technical sleight-of-hand is going on and the error you are seeing is not the real problem. Maybe it’s all an illusion…

Women in Tech Festival Global 2021

If you have worked in the tech industry for some time, you are likely to have noticed the issue with diversity. Information Technology was probably thought of as a male domain, and we can see the consequences of such thinking on a global level now.

67 Bricks strives to be a diverse and inclusive workplace, and we continuously improve our D&I awareness and practices. That is why for the second year in a row we attended the Women in Tech Festival aimed to champion diversity and empower companies and individuals to be allies for underrepresented groups. I did a presentation titled “It’s good to give back” at the event, which I immensely enjoyed, because, as a woman in tech myself, the topic of diversity is very close to my heart, and I take great interest in it.

This blog post gives a summary of some sessions I attended virtually.

Opening Note

The event started with the opening note from the Belonging, Inclusion and Diversity Lead of Investec, Zandi Nkhata. She spoke about reasons why women leave the tech industry: 

  • the lack of female role models.
  • experience of microaggressions – that is things people say to you that kind of remind you that you do not belong.  
  • the fact that your experience at a company might vary on whether or not you have an inclusive leader.

She also explained the difference between diversity and inclusion which I think is excellent: diversity is inviting someone to a party and inclusion is asking someone to dance. She also highlighted that only 20% of the workforce in the industry are women.

What can companies do to make their places diverse and inclusive? As an example, Investec’s vision is to make it a place where it is easy for people to be themselves, and to achieve that they set up different networks for people to speak up and listen to their feedback, provide learning and training opportunities about bullying, harassment and discrimination and have an allies programme.

Zandi also mentioned that it’s good to set KPIs with regards to diversity and inclusion, but they are not quotas, you have to be fair in achieving these targets.

Glass Ceiling or Sticky Floor

This panel discussion was about career progression – either knowing you’re probably the best candidate for a promotion yet not getting this promotion, or being capable enough but being obstructed by impostor syndrome, not having a career plan or a mentor.

The main point of the discussion was that a person finding themselves not progressing needs to ask themselves: “what is limiting my growth and what is in my control?”. You need to create a career plan and ensure you are in control. The importance of networking for women was highlighted, and events like Women in Tech is a great opportunity to do that. 

Another piece of advice was to focus on progress rather than perfection, and to learn to not be scared of asking questions even if you might think they are stupid (because they are not!).

Employers also have a duty to help with career progression. It is important to create career paths, understand them and enable employees to understand them as well, making it clear what is needed to get from A to B. It’s also vital to identify the strengths of each individual and know the exact purpose of each person in a team.

The panel also spoke about those who are in search of a new job and what question they might want to ask potential employers to decide about the suitability of a company for them; the suggestions were to look at the leadership gender balance and whether the company is doing any work regarding diversity and inclusion, among others.

Companies should not be scared to bring in people outside of the tech industry, reskill them and tap into their wealth of experience and transferable skills because the mixture of these experiences, strengths and insights can enable the team to grow.

How Old Are You?

This was about progressing in your career when you’re older. A lot of the focus here was on menopause awareness. This topic is still taboo, so safe spaces need to be created to make this conversation more visible, allowing people to speak about it without embarrassment. A lot of people still don’t know much about it even though their female relatives or friends might be experiencing menopause.

The speaker suggests that companies start with things like short talks about it in staff meetings. Some employers hold regular menopause cafes, others hold sessions on what to expect during this challenging time.  An emphasis was made on educating men (especially line managers) to feel comfortable about discussing menopause, and strategies for coping with it in all-male environments, which was mainly to push towards diversity and inclusion, having company policies around menopause and working together.

You Do Belong Here

This session was focused on combating impostor syndrome. This is typically associated with women (men do experience it too though) and the panellists shared useful tips that help them to overcome it:

  • Try to understand if it’s impostor syndrome or the culture that doesn’t let you grow. Some level of self-doubt is experienced by everyone.
  • Having a conversation about it helps combat it. It is the manager’s responsibility to create space where people can discuss it.
  • Some people use journaling. For example, you made a mistake, and 5 days later it’s still eating at you, and you still think about what you could have done. So to avoid that, by writing down what happened and what you could do next time, you get it out of your system.
  • Keep a list of your successes to read from time to time.
  • Refresh your CV and bio regularly as it allows you to focus on your achievements.
  • Educate yourself in neuroscience; humans are programmed to think negatively, and understanding this enables you to interpret your behaviour and thoughts.
  • Instead of changing yourself and trying to adopt a new personality type in certain situations to suit someone else, decide for yourself how you want to come across. However, beware if you go too far, If you’re not genuine, it’s not a sensible place to be. The best thing is to be your authentic self.

To sum up, thanks to this year and last year’s events I spoke to several inspiring females and got a bigger picture of what issues exist for women and LGBTQ+ communities in the tech industry, and what we can do to deal with them. I definitely learnt a lot from the Women in Tech Festival 2021. It was also great to realise that we, 67 Bricks, are doing all the right things to be as diverse and inclusive a workplace as we can. I look forward to sharing more of my learnings with my colleagues.

The First 67 Bricks Architectural Kata

On the 13th of October, 67 Bricks held its first Architectural Kata as an opportunity for developers to practice and experience architectural design. Ted Neward, the original creator of architectural katas, puts forward the argument for them quite succinctly:

  “So how are we supposed to get great architects, if

  they only get the chance to architect fewer than

  a half-dozen times in their career?”

Ted Neward

The kata exercise was quite simple to prepare for and run. Beforehand I picked a number of interesting katas from Neal Ford’s list, suitably anglicised them and prepared handouts for both online and in-person teams. Then on the day, it was a case of assigning everyone into teams and explaining the steps we’d be going through that day:

  1. Gather into teams and read through the assigned case study
  2. Discuss for the next 50 minutes and come up with an architecture, making sure to document any assumptions. Teams could ask me questions about the case study and I’d happily provide extra detail
  3. Each team would then take turns presenting their architecture to the other teams and fielding questions from them
  4. Finally a voting phase where everyone else gets the chance to show thumbs up/down/sideways to indicate how well they thought the presenting team did

Drivers

As a technical consultancy, we are often involved in architectural design of systems we create or take part in creating. Each client is individual; they seek different opportunities, are subject to different constraints and have different technical strategies. It’s critical for 67 Bricks’ success to have skilled architects able to influence and design suitable systems. 

Though, how can we develop these skills? As Ted Neward identified, most only get a half-dozen tries at it over their career. While books, online courses and talks help, knowledge needs to be applied and feedback loops closed to truly improve. We’ve never tried an event like a Kata before and I was interested to see what we could learn from it.

Sharing ideas and learning from one another effectively can be a challenge for any organization and made all the harder thanks to COVID-19 prompting a rapid move to remote working. The exercise looked to provide a great opportunity for participants to meet and work with others they may not otherwise have the chance to.

Kata Afternoon

Initially I hoped to get some of the more experienced technical leads to kick off the afternoon by talking about how they architect systems. This proved difficult. Those who I spoke to either thought they didn’t know that much or didn’t feel like they could speak to an audience well about how to do architecture. As such these initial talks didn’t happen and raised a lot of questions around how to arm participants with enough knowhow to feel comfortable tackling the task. I fear this may have led to some teams struggling too much to learn and make effective progress.

Splitting teams took some thought, I wanted to be sure each team had a mix of experience and tried to aim for teams of individuals who haven’t worked together before. I’m glad I spent the time to do this, the Kata would not have had the same impact if everyone was in the same team they usually are (one of the Kata rules is to try and break up regular teams).

Running the actual Kata was reasonably straightforward, if a little awkward thanks to having to manage both online and an in-person group. Some online teams got stuck waiting for me to join their Zoom room to answer their questions. While the in-person group was easy to recognise if they needed a question answered or a nudge in the right direction. 

I tried to prepare some broader scope for each case study before the event in preparation for questions being asked, I quickly found myself having to improvise. I actually found this surprisingly fun. Some teams may have made things harder for themselves by asking too many detailed questions and trying to cover every single detail in their architecture.

The presentation phase had mixed results, some teams did a strong job and were able to present their ideas well, answer questions clearly and came up with really suitable architectures. Other teams struggled a bit more, both with coming up with a suitable architecture and being able to communicate it with everyone else.

Feedback and Lessons Learned

Feedback was overwhelmingly positive, lots of participants really enjoyed the experience. They liked working with different people and felt they learned a lot from one another about how they tackle this type of task.

I think aiming for mixed team compositions was good and allowed individuals to have interactions with colleagues they wouldn’t normally work with. I’ve had feedback a number of times that with the move to remote, developers felt a lot more insular in their teams and I hoped this gave a chance to break out of those groups.

Doing both in-person and remote teams did make running the Kata a little challenging. I felt that I couldn’t effectively keep an eye on all the teams making it tough to know when to nudge a team that may be getting a little stuck or provide an answer to a pressing question.

Next time I would look at including publishing consultants to act as clients. I perhaps enjoyed being a fickle client a little too much. Having a more dedicated individual to represent the client creates opportunities for encouraging more of a discussion, simulating something closer to the real world when we work with our clients. It would also make it easier to manage the event and our publishing consultants can gain some insight to the architecture process. 

Conclusions

Given the positive response, I would happily organise another Kata in the future. If you are thinking about running one for your own organisation, I would highly recommend it. It’s a great chance to meet and learn from other devs. I certainly learned a lot simply observing the various teams taking different approaches to their case studies and the variety of solutions they came up with.

What have I been listening to?

A while ago, Tim suggested we could have a #now-listening channel in our company Slack, in which people could post details of what they were listening to. It occurred to me that it might be a fun challenge to try to figure out from what I’d posted on there who my favourite artist was, and which was my most-listened-to album. So I rolled up my sleeves and got to work. This is an account of what I did and my various thought processes as I went along…

Challenge: figure out how to get my posts from our #now-listening channel and do some statistics to them.

Continue reading “What have I been listening to?”

The Trials and Tribulations of a Working Parent

I’ve worked in the science and technology sector for my whole career, starting off by completing a PhD in Physics, then migrating into computer modelling, and then into software development.  I had my son during the final year of my PhD (oh how naïve I was about how easy that would be) and then immediately hit the dilemma of “how are we going to pay for this?”

Students are not entitled to any kind of maternity leave or pay – when I made enquiries about this I was advised to quit my studies, which would have made me eligible for various benefits.  My department was much more accommodating and gave me four months paid maternity leave – something they were under no obligation to do.  I also managed to claim Maternity Allowance  (https://www.gov.uk/maternity-allowance/how-to-claim), because I had done some part-time maths lecturing which made me eligible.

When I made my first forays into real paid employment, I had to tackle the thorny issue of childcare.  Childminder or nursery?  Enlisting the grand-parents wasn’t an option, and neither was either myself or my husband becoming a stay-at-home parent.  We initially picked a nursery at my husband’s work-place, and then found one closer to home.  The fees were astronomical – larger than the mortgage – but we scraped by.  I had some comments (addressed to me, never to my husband) along the lines of “why bother having kids if you’re never going to see them?” which I shrugged off.

We timed having our second child so that we would not have two children in nursery at the same time for long – if nursery fees for one child were astronomical then double nursery fees were on a whole other level.  My employer at the time (not 67 Bricks, I should add) only offered the barest minimum maternity package, and so I could only afford to take four months’ maternity leave (my take-home pay went down to around £120 a week after the first 6-weeks of leave).  This was before the change in the law that would have allowed my husband to take some extended leave himself – when he requested to do so his employer said something along the lines of “you can do that when men start giving birth to babies”.

Despite the deficiencies in that company’s maternity policy, my immediate line manager was wonderfully accommodating.  He allowed me to have an arrangement where I worked in the office for about 5 hours a day and then completed the rest of my hours at home.  This enabled me to reduce the nursery hours from 9-4 rather than 8-6, and save money while spending more time with the kids.  He also gave me considerable flexibility around school holidays, and working from home on days when I had various child-related errands such as school plays, parent appointments etc.

Speaking of school, nobody ever tells you that having a school-age child is actually harder to fit a job around than having a nursery age child.  The school day ends at 3pm – and who is ready to finish work at 3pm???  Also, there are 13 weeks a year of school holidays to somehow cover.  At the time, the school did not have an after-school club (they started one up in later years) and so I had to find a child-minder who could do after-school pickups.  There was a holiday club at a school in the next village which I used – it was heavily sports-related, which my son in particular did not like, but I told him there was no choice and he had to go.

Over the next decade I went through various child-care arrangements, including nursery, childminders, after-school clubs, holiday clubs, and various forms of flexible working arrangement for both myself and my husband.  Our days were organised with military precision.  Drop the kids at school at 8:40.  Drive like a maniac to work, never being able to arrive earlier than 9:30.  And then having to leave at 5pm on the dot so that I could once more drive like a maniac to pick the kids up by 6pm or risk being fined (typically an immediate fine of £15 and then £10 for every extra 15 minutes you were late).  And the stress of sitting in a traffic jam on the motorway, watching the clock tick, wondering exactly how late I would be.  I changed jobs and my new line manager was equally wonderful – I was never quite able to complete my full hours during office-time, but he was perfectly fine with me making up time in the evenings.  And he never once quibbled when I said that I could not get in any earlier than 9:30, or stay any later than 5pm.

My kids are now in secondary school and make their own way to and from school, and are old enough to be left alone during school holidays.  Therefore I finally no longer have to worry about pick-ups, or astronomical childcare fees, and I don’t have to rush around like a loon trying to pick them up by a specific time.

The key takeaways from all this are:

  • Parents need flexibility.  We have schedules to meet, parents evenings and school plays to attend, and sometimes sick children to tend to.  Having an understanding manager who doesn’t watch the clock, and allows you to complete your working hours according to whatever pattern works, makes our lives so much easier.
  • Child-rearing is expensive.  Attractive maternity packages will improve your staff retention and employee satisfaction no end. The 67 Bricks maternity policy is better than many (Employees who have been here 2 years get 12 weeks on full pay and 12 weeks on half pay).
  • Dads need flexibility too.  For every dad who you allow to leave early to do a school run, there is probably a grateful mum who is able to get on with her own job without worry.  The number of dads standing at the school gate is getting bigger year-on-year.  When I was a child my dad was the only one at the gate, but these days it is much higher and that is only a good thing.  I have seen male colleagues experience discrimination in previous jobs, for example expressions of incredulity when they state an intention to take paternity leave.

Sharing Failures Dev Forum: The most “interesting” bugs you’ve caused

Ely (n.)

The first, tiniest inkling you get that something, somewhere, has gone terribly wrong.

The Deeper Meaning of Liff – Douglas Adams & John Lloyd (Thanks to Stephen for pointing this quote out)

At 67 Bricks we value open and honest communication, and aim to treat each other with respect and humility. Normalising failure helps everyone understand that mistakes happen and shouldn’t be used as a mechanism to punish but instead are an opportunity to learn and become more resilient to future failures. This week’s Dev Forum was on the topic of sharing failures, specifically talking about the most “interesting” bugs you have caused.

64 Bit vs 32 Bit Ubuntu

Our first fail came from a developer working on converting data from base 64 as part of a side project. Developing on the 64 bit version of Ubuntu, the code was stable and the tests were green and so they continued to build the application out. The application was intended to work on a number of different environments including 32 bit systems and this is where the first problems emerged.

The 64 bit OS zeroed out data after it was read and the code was designed to work with this, the 32 bit OS didn’t and instead exhibited random test failures. Sometimes expected values would match up and other times it would be wrong by a character or two, typically at the end of the strings. Finding the root cause required a trial and error approach to eliminate possibilities and eventually get a grasp on the problem and come up with a solution.

Transactions and Logging

Working on a client application this developer made two perfectly reasonable design decisions:

  1. Use transactions with the SQL database to ensure integrity of data changes
  2. Write logs to a table to ensure they are persisted in a convenient location

This approach was perfectly fine while the system worked, but the mystery began when the client started to complain about errors with the system. The logs didn’t show any error messages that lined up with what the client was reporting. Why? The same transaction was used to store logs and to update the data in SQL. If an exception was thrown, the transaction was rolled back preventing any data corruption problems, but also rolled back all the logs!

The application was changed to use a different transaction for logging to ensure logs were persisted. Using these logs meant the root cause of the client problem could be resolved and a lesson learnt about unintended consequences.

Overlogging

Another mystery around missing logs. A web application used a separate log shipping application to take logs from the server and send them to a remote server. However, under heavy loads the logs would become spotty and clear gaps appeared. The reason for this was due to the sheer volume of logs the shipper had to deal with eventually becoming too great and causing it to crash.

The solution was to reduce the number of logs written so the shipper would continue to function even when the main application was under heavy load. This triggered an interesting conversation at the dev forum on the ideas of how many logs should be written. Should you write everything including debug level logs to help with debugging faulty systems? Or should you write no logs whatsoever and only enable them when something starts going wrong?

Naturally we seemed to settle somewhere in the middle, but there were disagreements. Possibly a future dev forum topic?

The Wrong Emails

A client needed to receive important data daily via fax and the application did this by sending an email from a VB app to an address which would convert the body to a fax and send it on. The application needed testing and so had a separate test environment that was presumed to be safe to use for running tests.

It turned out that the test system was using the same email address as the live system, meaning the client was receiving both test and live data. Luckily the client caught on and asked why they were receiving two emails a day and the test system was quickly updated.

From then on this developer now always uses some form of override mechanism to prevent sending actual emails to end users. Others use apps like smtp4dev which will never send on emails (and just capture them so they can be inspected) or services like SES which can be configured to run in a sandbox mode.

Hidden Database Costs

AWS can be very costly, especially if no one is watching the monthly spend. In this case, one database drove up costs to astronomical new highs due to a single lambda running regularly and reading the entire database. The lambda was supposed to delete old data, but had a bug which meant it never deleted anything. Several months of operating the system meant a lot of data had piled up and so did the costs. The fix was simple enough to apply and we added additional monitoring to catch cases of runaway costs.

Similar experiences have been had before with runaway costs on clustered databases like MarkLogic. Normally a well-built index will be able to efficiently query the right data on the right node, but if the index is missing, MarkLogic will pull all the data across from the other node to evaluate it. This can drive up some eye-wateringly high data transfer prices in AWS and as such we now always monitor data transfer costs to ensure we haven’t missed an index.

Caching Fun

Our next issue is about users appearing to be logged in as other users. The system in question used CloudFront CDN to reduce server load and response times. The CDN differentiated between authenticated and unauthenticated users so different pages could be served for users who were authenticated.

The system made use of various headers set by lambdas to differentiate between authenticated and unauthenticated users. The problem arose when session handling was changed and the identifier used was accidentally stored within the CDN. This caused an issue where a user could load a page with a set-cookie header that set the identifier used for a different user’s session.

The team solved this bug by tweaking the edge lambdas to ensure only non-user specific data was cached. Caching in authenticated contexts can be challenging and need to be very carefully considered how they will be used reliably.

Deactivate / Activate

In this bug the business asked for a feature where user accounts could have a deactivated time set in the future that when reached, the user would be considered inactive and unable to access the system. This feature was implemented with a computed field in the SQL server which could be used to determine if the user is active or not.

As the system was already in use, migration scripts were developed to update the database. These needed to be applied in a certain order, however, deployment practices for this system meant that someone else applied the scripts and ended up causing an error where all users ended up deactivated preventing any users from accessing the system. To restore service, the database was rolled back and ultimately the feature was abandoned as too risky by the business.

Some viewed this as an example of why services should be responsible for their own database state and handle database migrations automatically.

Creating Bugs for Google to Index

Magic links can be a very useful feature for making authenticated access to a website easy, as long as these urls remain private to the correct user. In one case the url got cached by Google including the authentication token meaning anyone who could find the link would be able to access the authenticated content! This was fixed by asking Google to remove the URL, invalidating the token in the database and ensuring metadata was added to appropriate pages to prevent bots from indexing pages.

Another Google based bug next; after building a new system, the old one needed to be retired and part of this involved setting up permanent redirects to the new system. However, Google continues to serve up the old site’s urls as opposed to the new system’s urls, and a fix is still being worked out. A lesson learned on how important it is to carefully consider how search engines will crawl and store web sites.

As we see, failures can come from any number of sources, including ourselves. Bugs are a perfectly normal occurrence and working through them is an unavoidable part of building a robust system. By not fearing them, we can become more adept at fixing them and build better systems as a result.

Session musician programmers

As a software consultancy, we are always in the business of trying to recruit good developers. One of the more annoying phrases that has cropped up in the industry lexicon of late is “rockstar programmer”, as both an ideal to which developers are assumed to aspire, and a glib description of the kind of programmer that software publishers are assumed to be desperate to employ.

Continue reading “Session musician programmers”

Dev meeting – debugging XQuery and XSLT – 8th January 2021

In the dev meeting today, we talked about debugging XQuery.

XQuery is a query language (primarily) for XML (in very loose terms, it’s like SQL for XML). Reece is the developer of an IntelliJ IDEA plugin that supports development of XQuery in IDEA. The latest changes he’s made to it allow it to be used for debugging XQuery in MarkLogic.

In IDEA, after installing his plugin, you need to set up an XQuery Run/Debug configuration for the MarkLogic server you are debugging. Then, you can run an XQuery file from within IDEA against MarkLogic, and it will display the output in the MarkLogic console.

Using standard IDEA breakpoints, you can add a breakpoint to an XQuery expression. Then, you can debug, which shows the stack frame, and the current variables and their values.

There is some complexity in working out the stack frames for eval expressions, which dynamically execute XQuery code in strings. Inigo expressed the opinion that this was generally a bad idea anyway.

At the moment, it’s not possible to debug into X-Ray XQuery tests, but Reece is doing further work to make it possible to run X-Ray tests easily from IDEA, which would also include work to make X-Ray tests debuggable.

Reece is also working on debug support for Saxon, which will support debugging in XQuery and in XSLT via Saxon.

Clarity and XQuery – dev meeting fun with blockchains and Intellij IDEA

Alex talked about Clarity, a language for running on the Blockstack blockchain. He recently worked on a hackathon for it. One of the motivations was to improve his functional programming – Clarity is very Lisp-like, and so being forced into a functional mindset was useful for learning. Another motivation was that Clarity were offering cash prizes. Clarity is pretty new – and things like the test framework and code syntax highlighting are very new.

Alex wrote a high score system that runs in Clarity – that will maintain the high scores for a game long-term, rather than needing to be maintained by a particular game developer on their own server.

There are various limitations to Clarity because of the nature of running on the blockchain – like an inability to perform loops, because it’s not possible to prove that they resolve in a reasonable length of time, due to the halting problem. And some primitives cost money, like storing and retrieving variables in the blockchain.

There is a built-in testing framework, that allows you to test your code on your local machine without writing to the blockchain. It’s incomplete at the moment, because it does not support transactions.

Native lists cannot be unbounded, for similar reasons of cost. So Alex also wrote an “endless list” that is essentially a linked list of bounded lists, so you can treat it as an unbounded list but you are only acting on the portions of list that you are currently looking at.

He then wrote code that can be used for redistributing money from a group of donors to a large group without having to use existing infrastructure like Paypal – because organizations like Paypal don’t like it when individuals send money to lots of people. This was inspired by some work that was done based on US stimulus cheques – individuals got cheques from the government, but not all of them needed them, and plenty of people needed more than the stimulus cheque provided. This allowed a group of donors to store money in the contract itself, in a way that is transparent to everyone; and then it to be distributed out to a set of recipients.

There’s more information at https://community.blockstack.org/clarity-winners.

——–

Reece talked about the latest improvements to his IntelliJ IDEA XQuery plugin. It now shows inline parameter names on function calls, and shows the function structure. It also displays documentation for functions from MarkLogic or from the standard XQuery and XPath libraries – it will download the documentation.

It will also show a query log in a tab for the various MarkLogic log files, so you can see the latest log messages without viewing a separate file. This works for both local log files, and for remote MarkLogic servers.

It will parse MarkLogic rewriter XML files, and display the results as a list of REST endpoints. In the upcoming version, it will automatically retrieve all the registered MarkLogic schemas, so for the rewriter.xml it provides auto-complete for the various structural elements and options in the rewriter XML. It also parses RestXQ APIs as used in systems like BaseX.

It will handle relative imports correctly if you set an XQuery directory as a source root – so “/myLibraries/whatever.xqy” will resolve to the right place. In the upcoming version, it will be able to recognize these automatically.

When you’re executing queries locally using a configured query processor (e.g. if you’ve configured MarkLogic or another system so you can do the local equivalent of using the MarkLogic console), then you can pretty-print the output and you can also see profiling information for the code that you execute.

Reece is currently working on debug support, so you can set breakpoints in your XQuery code.

Inigo stated that Reece’s plugin is great, and that everyone using XQuery or MarkLogic should be using it.