Sending monetary amounts over network

If you write any application that operates on money – even a simple web shop, sooner or later you will have to send monetary amounts through API calls, SQL queries, CSV downloads, and so on. Let’s see how to avoid common mistakes which could destroy your business!

Money over an API

Whether you request a payment via an external API, or expose your own data to your clients, it is important to set up a protocol. The most popular data format for API calls is JSON. However, this format accepts mostly anything you’ll feed it with, so it’s not going to magically guess that you’re sending financial data – like the product info below:

{
  "id": 1,
  "name": "Keyboard",
  "price": 22.99,
  "availability": "in stock"
}

What does 22.99 mean here? What is the currency? Are you aware that you’re using a float data type here, which makes the amount imprecise? And if there was an integer like 23is it an amount of dollars or cents?

The above piece of JSON would look a lot better like this:

{
  "id": 1,
  "name": "Keyboard",
  "price": {
    "amountDecimal": "22.99",
    "currency": "USD"
  },
  "availability": "in stock"
}

Of course instead of amountDecimal you can use amountInteger if you prefer. The key is to let everyone know what exactly that value means!

Money in a SQL database

Unfortunately, SQL databases aren’t helpful with storing monetary data, so you have to take care of it by yourself.

There are two solutions:

  1. Use one BIGINT or DECIMAL/NUMBER column to store the amount, and a CHAR(3) column to store currency code. This way you’ll be able to sort by price or do some calculations directly in SQL queries, however be careful not to mix different currencies.
  2. Use one VARCHAR column to encode the whole value, like USD 123 or USD 1.23 (depends if you prefer using cents or dollars).

Never use the FLOAT type which is not suited for precise decimal calculations!

Money in a CSV file

This one can be done horribly wrong without proper encoding. CSV stands for Comma-Separated Values. An example CSV file containing a list of products can look like this:

ID,Name,Price,Availability
1,Keyboard,22.99,in stock
2,Mouse,5.99,out of stock

The comma is a special character here which is used to separate columns. However in some languages, comma acts as a decimal separator, so you would have 22,99 instead of 22.99. This would break the table while parsing!

You can solve this issue by using a dedicated CSV library for data export. Any values containing a comma will be automatically encoded. Usually the whole value is enclosed with quotes or the comma inside a value is doubled.

Two more questions arise:

  1. What is the currency in the file above? Do these numbers refer to dollars, euros, or something else? There should be a separate column specifying the currency!
  2. While parsing, how do we know what number format was used? Should we expect a dot or a comma inside a number? What language rules should we use?

Take care of your data!

When exchanging financial data with other systems, always make sure to:

  1. Use a proper unit: either dollars or cents, either euros or cents, and so on.
  2. Never use floating-point numbers. Send a string like "19.99", not just 19.99. Or use integers.
  3. Specify currency using the ISO 4217 standard.
  4. Respect the encoding rules of the data format you’re using.
  5. Parse localized money strings properly.

Formatting monetary amounts with Java Money

Java is very often used to develop big financial systems, yet still the JDK has little or no support for monetary arithmetics, currency conversion, formatting money for different locales etc. There’s only a Currency class which serves as a list of ISO 4217 currencies.

Fortunately we’ve got a JSR-354 standard and Moneta – the reference implementation. It provides us the Martin Fowler’s money pattern along with string formatting and currency repositories.

In this article I will focus on displaying monetary amounts for different currencies, locales and personal preferences.

Basic example: amount and ISO symbol

final Locale[] locales = {
    Locale.US,
    new Locale("nl", "NL"),
    new Locale("pl", "PL")
};

final Money amount = Money.of(12345.67, "USD");

for (Locale locale : locales) {
  MonetaryAmountFormat formatter = MonetaryFormats.getAmountFormat(locale);
  System.out.println(formatter.format(amount));
}
en_USnl_NLpl_PL
USD12,345.67USD 12.345,6712 345,67 USD

Amount with a currency symbol

final Locale[] locales = {
    Locale.US,
    new Locale("nl", "NL"),
    new Locale("pl", "PL")
};

final Money amount = Money.of(12345.67, "USD");

for (Locale locale : locales) {
  MonetaryAmountFormat formatter = MonetaryFormats.getAmountFormat(
      AmountFormatQueryBuilder.of(locale)
          .set(CurrencyStyle.SYMBOL)
          .build());
  System.out.println(formatter.format(amount));
}
en_USnl_NLpl_PL
$12,345.67USD 12.345,6712 345,67 USD

The $ currency symbol is displayed only for the United States locale. Other languages do not use it.

Amount with a full currency name

final Locale[] locales = {
    Locale.US,
    new Locale("nl", "NL"),
    new Locale("pl", "PL")
};

final Money amount = Money.of(12345.67, "USD");

for (Locale locale : locales) {
  MonetaryAmountFormat formatter = MonetaryFormats.getAmountFormat(
      AmountFormatQueryBuilder.of(locale)
          .set(CurrencyStyle.NAME)
          .build());
  System.out.println(formatter.format(amount));
}
en_USnl_NLpl_PL
US Dollar12,345.67US Dollar 12.345,6712 345,67 US Dollar

Amount without a thousands separator

final Locale[] locales = {
    Locale.US,
    new Locale("nl", "NL"),
    new Locale("pl", "PL")
};

final Money amount = Money.of(12345.67, "USD");

for (Locale locale : locales) {
  MonetaryAmountFormat formatter = MonetaryFormats.getAmountFormat(
      AmountFormatQueryBuilder.of(locale)
          .set(AmountFormatParams.GROUPING_SIZES, new int[]{2, 0})
          .build());
  System.out.println(formatter.format(amount));
}
en_USnl_NLpl_PL
USD12345.67USD 12345,6712345,67 USD

Grouping sizes are specified starting from the decimal point. In the above example we have two digits after the decimal point, and then we have no more grouping going to the left. Of course other combinations are possible, e.g. {2, 3, 4}.

Custom pattern

Using a pattern as defined by java.text.DecimalFormat.

final Locale[] locales = {
    Locale.US,
    new Locale("nl", "NL"),
    new Locale("pl", "PL")
};
final Money amount = Money.of(12345.67, "USD");
for (Locale locale : locales) {
  MonetaryAmountFormat formatter = MonetaryFormats.getAmountFormat(
      AmountFormatQueryBuilder.of(locale)
          .set(AmountFormatParams.PATTERN, "###,###.## ¤")
          .build());
  System.out.println(formatter.format(amount));
}
en_USnl_NLpl_PL
12,345.67 USD12.345,67 USD12 345,67 USD

Using your own currency in MoneyPHP

I have this talk about the MoneyPHP library and handling currencies in general. I often get a question: is it possible to define custom currencies in MoneyPHP? A possible use case would be loyalty cards for example, with their own scoring system.

The answer is: YES! MoneyPHP not only accepts all ISO 4217 currencies, but also allows you to define a custom list with any identifiers and any subunits. So for example, you can create a currency called BlahBlahBlah that has 7 decimal points and then convert an amount from your currency to another by providing a custom exchange list.

In the example below, we’re creating two fictious currencies: MyPoints and TheirPoints. The first one has two decimal points while another has none (only whole numbers). We create a Money object that holds exactly 123.45 MyPoints.

Then we specify an exchange table which says that 1 TheirPoint equals 0.5 MyPoints. We create a currency converter and then try to convert one currency to another. 123.45 * 0.5 equals 61.725. But since TheirPoints subunit is 0, the amount has to be rounded up, so we get 62.

use Money\Converter;
use Money\Currencies\CurrencyList;
use Money\Currency;
use Money\Exchange\FixedExchange;
use Money\Money;

$currencyList = new CurrencyList([
    'MyPoints' => 2,
    'TheirPoints' => 0,
]);

$myCardAmount = new Money(12345, new Currency('MyPoints'));

$exchange = new FixedExchange([
    'MyPoints' => [
        'TheirPoints' => 0.5,
    ],
]);
$converter = new Converter($currencyList, $exchange);

$theirCardAmount = $converter->convert(
    $myCardAmount, new Currency('TheirPoints')
);

echo $theirCardAmount->getAmount();  // 62

However, there can be a problem with currency formatting using PHP’s intl library. Consider this example:

$numberFormatter = new \NumberFormatter(
    'en_US', \NumberFormatter::CURRENCY
);
$moneyFormatter = new \Money\Formatter\IntlMoneyFormatter(
    $numberFormatter, $currencyList
);

echo $moneyFormatter->format($theirCardAmount);  // The 62.00

The odd output (The instead of Their and the unwanted fractional part) comes from the fact that PHP’s NumberFormatter handles only ISO currencies which have 3-character symbols. So you should use \NumberFormatter::DECIMAL instead and add your custom symbol manualy.

See my post about handling money in PHP

Doing Java the bad way: loose typing

In the late 90s, people loved PHP because it was so easy to use. And one of the reasons for that was loose typing. You could quickly send any data to the browser without caring about data types.

This became a problem when developers started to build bigger and bigger applications with PHP. Strict typing became an important feature – first introduced in HHVM and later adopted by PHP 7. However, there were countless discussions on PHP internals list on whether to enforce strict typing or not. In the end, strict typing became a feature on request. And it didn’t work for variables or class properties (the latter will change with PHP 7.4).

What I always liked about Java was strict typing everywhere from day one. Java code screams to me, like “hey, I got a product object of type Product and this object cannot be anything else by accident”.

Later I saw that you can make a big mess with every technology, even Java.

Every Java object inherits from the Object type. And you can even create hashmaps of anything that derives from Object. So these maps can even contain further maps of every possible creation.

It is sometimes tempting to code Java the PHP way. It’s quick. You can have one box for every kind of stuff. But then it becomes a nightmare.

First warning sign of an antipattern:

Map output = new HashMap<>();
output.put("stuff", stuff);
output.put("otherStuff", otherStuff);
response.setData(output);

Second warning sign:

public void setData(Object data) {
    // Oh, so I can put any data here...?
}

Third warning sign – the tests:

Map output = (Map) response.getData();
// now we get an "unchecked warning" and we have no idea what's inside "output"

Try to answer this tricky question first: What does my web app do? You could say: Well, it retrieves data from the database and sends them to the user.

Wrong answer. How about this one: My app retrieves products and orders from the database and sends them to the user.

If you put it that way, you’ll begin to understand that you need classes like Product, ProductsList, ProductsListRequest, ProductsListResponse and so on. Well, maybe not all of them (don’t over-engineer!), but at least don’t put everything possible into the data bins.

Every HTTP request and response in your application must have a specified structure anyway, so try to formalize it by creating proper classes.

Why you should disable wildcard imports in IntelliJ IDEA

The default behavior of IntelliJ IDEA is to replace multiple class imports from a package with a star/wildcard/asterisk. So for example when you’ve got:

import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;

…and you try to import another class from java.util, everything is merged into one mass import: java.util.*.

At my team, we decided to turn that behavior off. Why and how?

  1. It is bad for version control. Suppose one developer is using Eclipse and just adding another line to the imports. Then another developer is editing the same file and merging all the imports. We’ve got a merge conflict here! Also, with star imports, it’s harder to track what has been added to the local file namespace.
  2. It can lead to unnecessary naming collisions. There can be a lot of User classes in different packages. By importing everything from a package we can accidentally import something we do not want. Also, when a dependent package is upgraded, it might pollute our namespace with new imports that we’re not even aware of.

IntelliJ does not have a way to turn off star imports, but it has a threshold option. So everyone in the team set a ridiculously high threshold (like 500).

Here you can prevent IntelliJ from merging your imports

Another problem is folding imports by default. I believe that the imports section is actually a very important piece of code and it should not be folded. We need to be aware of what’s going on there, so we jumped into Settings | Editor | General | Code Folding and turned Imports off.

Further reading

https://www.jetbrains.com/help/idea/creating-and-optimizing-imports.html#disable-wildcard-imports

Why I didn’t end up being a database engineer

Once upon a time, looking for new career opportunities, I considered becoming a database engineer or architect. I was pretty successful in optimizing MySQL databases growing up to 65 GB. I liked it. However I noticed almost no one was hiring MySQL database engineers – it’s too simple. Real challenges await those familiar with Oracle and big finance systems, aviation etc. So, I bought a thick book about Oracle and… never used it.

The only profit I got from buying an Oracle guide was another book I was offered in package: “SQL Antipatterns: Avoiding the Pitfalls of Database Programming” by Bill Karwin. It provided me a lot of valuable tips which I still eagerly keep on sharing with my coworkers.

Is database the center of every app?

Anyway, I used to believe that a database is in the center of every application. Most of them process data, right? So I started all projects from conceiving a database model: relations, data types, queries, and so on. I spent a lot of time optimizing data structures.

My approach had several pros. First of all, it was intuitive for other developers because a good database model made them immediately understand the purpose of the system and navigate easily through the rising codebase. Secondly, a good model helped datasets grow significantly (like hundreds of millions of records) without performance loss. Moreover, with correct foreign keys and other constraints, data remained consistent.

I have a funny story about a team of developers which refrained from using foreign key constraints. They used ON DELETE CASCADE clause everywhere and they accidentally wiped away half of the test database. It was such a shock for them they removed all the foreign keys. Soon, data became terribly inconsistent and a lot of bugs surfaced in the apps depending on that database.

Carving a SQL giant

Optimizing databases was cool, but there was one bad practice I really disliked looking at some DBAs work. After I read about SOLID principles, Domain-Driven Design and other stuff, I started to hate SQL stored procedures, especially when they were too long.

I saw some MS SQL devs trying to handle all the business logic with SQL scripts. These systems already had many bugs, and my fellow devs just kept on adding more CASE…IF blocks after requests from the business people. After some time, it was nearly impossible to add or change anything without breaking stuff.

Such situations made me realize that you have to choose the right tools for the job, not the other way around. Of course data is a huge asset for every company, but behavior is important too. Is SQL a right tool to model complex business behaviors?

Getting to know the big picture

For me, data has never been the biggest challenge. It is always the sophisticated business logic that needs to be modelled very carefully. It evolves over time, and we have to make sure all developers (especially the new ones) dig it quickly and flawlessly.

I decided to improve my process modelling skills with OOP and FP, know SOLID, DDD and other widely adopted principles or design patterns. I decided to share business knowledge with all my teammates to make sure they understand the purpose of their work and nature of the surrounding business. Of course SQL knowledge is valuable too, but it’s not enough.

Is Portable Document Format obsolete?

I spent two years on a project which aimed to create highly sophisticated PDF reports for teachers. These automatically generated reports contained bar chars, pie charts, line charts, a lot of tables and everything else. It worked: end users were very happy about the new reports system. But still, something keeps bothering me.

Making PDFs can be tough. Do we still need them?

There are many ways you can generate PDF files. I started from a Java class which calculated every table cell dimensions, position of every text line etc. Then I used PHP libraries like FPDF and TCPDF. The latter had a very simple HTML parsing engine. Then I came across Zend_Pdf and kept using it until I stumbled upon wkhtmltopdf. Yeah, that was a blast! Someone put together a WebKit rendering engine and a PDF printer.

But even wkhtml was not prepared for all the challenges that my client had for me. Dynamic headers and footers. Sophisticated charts which I prepared as SVG files. Errors with dividing tables across multiple pages (with table headers of course). Lack of dynamic font size adjustment depending on the text amount. Trouble with applying custom fonts.

Chrome 59 offered printing to PDF via headless mode. Having recent Chrome rendering engine is great, but I lost some features available in wkhtmltopdf (like these crazy page headers and footers).

You can already see that generating proper PDF files might require a lot of tweaking and quirks. What if we don’t need PDFs at all? In the era of portable devices coming in all shapes and sizes, do we still have to limit our documents in an A4 or any other paper format?

Is PDF always a right choice for our project?

Let’s think about the business needs for a while. The system I mentioned was designed for teachers who got used to print everything on a daily basis. I’m not sure how it works in other countries, but Polish teachers print a lot of stuff. They download a PDF, then print and put on their desks to discuss with pupils and their parents. That’s their habit.

I worked on a similar reporting system designed specifically for parents. The initial business decision was to copy the same well-known PDF system and do some adjustments. I asked my client: hey, do we really need to spend a lot of time fine-tuning PDFs again? Do our end users need PDFs? They have different habits, they have smartphones and tablets. They are not going to print our reports. They want a modern, responsive web app.

That’s how I convinced my client to drop PDFs in the new system. I made development a lot faster and easier. We made a modern frontend layer using ReactJS and everyone was happy with the result.

Do we still need static PDF layouts for invoices, order confirmations, tickets, magazines, official documents, reports?

In Poland, we have electronic train tickets that can be downloaded on a smartphone. It is an ordinary PDF file with all the ticket details. But all we have to do is to show a QR code during ticket control. The same applies to concert tickets. Why use PDF if a simple HTML would be enough?

Further reading

Will PDF files become obsolete in 5-10 years?

There are some limits to the YAGNI principle

Have you heard of YAGNI? The acronym stands for “You Ain’t Gonna Need It“. The original meaning, as Extreme Programming guru Ron Jeffries said, was to “implement things when you actually need them, never when you just foresee that you need them”. But how did this principle evolve over time?

Since the late 90’s we have witnessed a lot of frameworks and tools emerge. We love new shiny tools, don’t we? Sometimes we think that a new framework will solve all our problems. That’s what the “Hype Driven Development” article described.

So then people started to talk: okay, maybe we should care more about the business, not just our fun? Maybe we don’t need so many tools and libraries everyone else is hyped about? Maybe we should not do a big rewrite of our systems every six months after another tech conference?

Don’t do everything by yourself

If using the latest hyped tool for every task is one extreme, then doing everything by yourself is another one. The latter option was proposed by some guy conducting a training on microservices (another hype term by the way) for my company.

I was in a team using PHP and we wanted to know something more about developing proper microservices architecture. We did a lot of analysis of our current system and decided this might be a chance. What we heard during the training was:

  • Our coach would never use Symfony for a microservice because “it’s huge and will cause performance issues”. That was a couple of months before Symfony 4 was released, but the same guy argued that “computing power is cheap and we shouldn’t care”. Umm…
  • Our coach would never use Doctrine for a microservice because “it’s huge and will cause performance issues”.

Instead of showing us some alternatives, the coach asked us to write an example application in vanilla PHP.

By the time the training was conducted, I had already spent years working on home-brew “frameworks”, made by people who did not believe in the popular systems and desperately wanted to do things their own way. Such people tend to leave the company a couple of years later, overwhelmed by all the issues caused by their “ingenious” frameworks.

Pick tools carefully, get to know them well

So you’re afraid of incorporating third-party code into your project? Cool, it means you’re a responsible developer. But before you turn down popular and well-tested solutions, get to know them well. How much can you fine-tune them? How much can they be customized and stripped of unnecessary features?

Back to PHP stuff, Symfony is highly customizable especially since version 4. Symfony and Doctrine tend to cache a lot of things. You can also optimize Composer’s autoloader.

Don’t optimize prematurely

Performance can be easily improved. But business logic can be a real monster.

After a couple of years working on big, sophisticated projects I realized that performance is a secondary issue. The biggest challenge for me has been always dealing with complex business logic, huge number of moving parts dependent on each other, unclear rules etc. You need good tools to model and test that logic.

So what you shortened page loading time from 300 to 100 milliseconds if your teammates do not understand the crazy optimizations you just made? What benefit do you have after forcing your team to use weird tools they will complain about?

You need to balance these things out.

Defensive coding: Null-safe string comparisons

One of the most valuable tips I received why having my Java code reviewed was to change this code:

if (merchant.getSalesChannel().equals("Mexico")) {
    …
}

to this:

if ("Mexico".equals(merchant.getSalesChannel()) {
    …
}

Everything in Java is an object. Thus, a simple string literal immediately creates a new object. We are sure that a literal is not null.

The other part of the equation might surprise us with a null. Who knows what that getter might return? Most likely it is some value from the database. Are we sure that the value was properly retrieved? Are we sure there’s a NOT NULL constraint on the database column? We can never be sure.

So if we want to compare a variable to a string literal, let’s use the null-safe style proposed in the second listing above.

Further reading:

https://stackoverflow.com/questions/43409500/compare-strings-avoiding-nullpointerexception/43409501

https://softwareengineering.stackexchange.com/questions/313673/is-use-abc-equalsmystring-instead-of-mystring-equalsabc-to-avoid-null-p

https://www.javacodegeeks.com/2012/06/avoid-null-pointer-exception-in-java.html

Defensive coding: Make local objects final

In the very first article on defensive coding we talked about avoiding mutability.

Let’s talk about JavaScript for a while. Every modern tutorial says that you can instantiate new objects either with let or const. You use let if you intend to create a variable (like with an obsolete var keyword), or you can use const to indicate that an object – once initialized – should remain constant. This gives developers great clarity and saves them from mistaken mutations.

What about Java? You don’t have a const keyword, but you have final. I love using it and many developers in my current project also use final wherever feasible. You can mark properties, local objects and method arguments as final. There are very rare cases when we need to mutate an object, so must of the time I end up having final everywhere:

public BigDecimal getTotalPrice(final Product product,
                                final BigInteger quantity) {
    final BigDecimal totalNetPrice = product
             .getUnitPrice()
             .multiply(quantity);

    return totalNetPrice.multiply(product.getTaxRate());
}

Why use final everywhere? It can look strange in the beginning, but it protects us from bugs. How many times have you seen robust methods spanning multiple screens, with vague and similar variable names? How many times have you mistaken foo, oldFoo, databaseFoo, foo1 etc.?

In PHP, the final keyword can be applied only to classes. You can’t create local constants, only class constants with const keyword. It’s even worse that you don’t clearly distingish first object instantiation from subsequent instantations using the same variable name. This code is perfectly valid in PHP:

$builder = new ProductBuilder();
$builder = new UserBuilder();