2016-09-26

Is the internet ready for DMARC with p=reject?

No.

DMARC is an email policy that builds on DKIM and SPF to provide a way for email senders to declare: "All email from this domain comes from this set of servers (SPF), and is signed using these public keys (DKIM)."

It's nice and well that we are finally able to do that, but DMARC comes with 3 modes of operation: p=reject, p=quarantine, and p=none. These modes suggest what a recipient should do if an incoming email doesn't match the sender domain's DMARC criteria, either due to signature failure (DKIM) or incorrect sending server (SPF). The default mode, p=none, boils down to "do whatever, maybe this helps you guess if the email is spam". The mode p=quarantine treats the email as spam, and p=reject is supposed to cause the message to be rejected at the point of ESMTP delivery.

Well, I tried p=reject, and it kinda works. Mostly. Except in the following situations:
  • If you send to any mailing lists from your domain. Your messages may be DKIM signed, but when the mail server forwards them, they will be sent by a mail server that does not have authority to send messages for your domain according to your SPF criteria. Message headers may also be modified so that the DKIM signature does not validate. This will cause your messages to the mailing list to be rejected by many subscribers at the point of ESMTP delivery. This in turn may cause the mailing list to frown on you.
  • If you send to anyone with a multi-layered mail server setup. In this case, your message is received correctly by the front mail server, which is configured to forward it to a back mail server. You cannot control this because it's a setup specific to the recipient and unknown to you in advance. But the back mail server is poorly configured to verify DMARC criteria, and rejects the message because it violates your SPF policy – from the perspective of the back mail server, it is coming from the wrong IP. The message may then be deleted silently, or sorted into trash.
This means you pretty much can't safely run a mail server with DMARC policy set to p=reject. For the same reasons that p=reject is not safe, p=quarantine also makes no sense. There's not much difference between causing your outgoing mail to be bounced and/or deleted, or causing it to land in spam.

DKIM and SPF are good ideas, but it's unfortunate that the policy has to be "none".

2016-09-08

Lock-step refactoring

This may seem obvious, but I am biased to avoid saying things that are obvious to me, and then things don't get said that are worth saying.

A nice thing about statically typed languages, like C++, is how you can use the compiler to help refactor.

Code often needs to be refactored. If it is not, it gets harder to fix issues and implement new features. When this happens, the temptation arises to throw the code away and write new code, but this is almost always a mistake. The correct process is to refactor code that works, to disentangle anything that needs disentangling, so that improvement becomes easier.

When there is existing code you can use, you should only ever write new code if you're willing to go through the entire years-long process of how it needs to work, and why it needs to work that way. Otherwise, use existing code, and keep improving it.

Refactoring is scary. It is scary because you have this hairy code, and you're concerned that any change you make to it will break it.

Behold, the beauty of a statically typed language. By leaning on the compiler, you can refactor in such a way that code will not compile until it is fully updated – and ideally, correct:
  1. Start with code that compiles and passes any unit tests you might have. Write unit tests if there aren't any. Focus on functionality you're worried about, and want to fix and/or preserve.
  2. For each modification you want to make:
    1. Find a way to make this modification so that the code will not compile unless all parts that need to take this modification into account are updated.
    2. Repeat the build as many times as necessary to identify all parts you need to update.
  3. When the code builds, if you used the compiler to aid you correctly, you should have found all the places that required updates. Unit tests should not fail because of missing an update. They should only fail if an update you made was incorrect.
For example:
  • If you change parameters accepted by a function, make sure no possible call using the previous signature will compile with the new signature.
  • If this cannot be achieved, rename the function, so that references to the old name will not build.
  • If you change the behavior of a function in a way that requires callers to update, change its signature in a breaking way, or change its name.
  • If you change the rules regarding a widely used data member, change its name.
  • If you change the rules regarding a widely used type, change its name.
Aspire to perform all changes in such a way that, if you're not 100% done, the code will not even compile. Don't make changes that allow the code to compile if you're only 1%, or 40%, or 99% done.

2016-08-28

Loyalty to group vs. principle

Principles, nearly by definition, are rules that when they are adhered to, lead to good results. Group loyalty, then – loyalty to specific preferred people over principle – necessarily produces worse results. The two are not the same, so to maintain loyalty to a group, one must depart from principle.

Countries differ on the ubiquity of group loyalty over principle. Prosperous countries tend to be more loyal to principle; shittier countries tend to have more loyalty to groups. It takes a prosperous environment to be able to afford to be loyal to principle, but loyalty to principle also makes a prosperous country. In comparison, people in shitty places feel like they need to be loyal to their families and local groups, but it's also the group loyalty – and lack of principle – that makes those places shitty.

You can find shitty subcultures where group loyalty dominates even in prosperous countries. Compare inner city ghettoes with the "snitches get stitches" mentality. Or the lifestyles of gypsies in Europe, whose loyalty is only to their group. Poor people often have high loyalty to family – because they must, their family is struggling – and this is what helps keep them poor.

To the extent that group loyalty exists among the elite – the rich, the police – it serves to bring down the rest of the country. Group loyalty among the rich results in an economy that benefits them, instead of most other people. Group loyalty among the police results in them mistreating, often killing people, and finding no wrongdoing among themselves.

It needs to be accepted as an ethical standard that people need to be loyal to principles. Loyalty to a group, ignoring principles, needs to be viewed as a form of corruption.

2016-08-24

Limitations of single-payer health care

The US runs a degenerate health care system, with abhorrent symptoms that are too many to enumerate. I offer a small tasting sample:
  • Drug companies that lobby to ban older inhalers whose patents are running out, so they can continue selling new patented ones at high markup.
  • Drug companies that lobby Congress to create demand for a product, then jack up prices to $609 per unit to deliver $1 worth of a drug.
  • Private insurance companies created as an accident of history, lobbying to perpetuate a system that routinely denies health care to people who most need it. Before the Affordable Care Act, they would deny coverage to people who they knew would be a net loss. Now, they offer poor people plans costing $1,000 per month, where they don't even cover anything under $5,000.
  • Health insurers may engage in shady practices. Just one, now deleted comment:
    [Insurance company] had my parents sign a bunch of stuff that absolved them from paying for my burn unit bills just days after [I] was burnt. My parents [don't] even remember signing them since they themselves were in shock.
  • Insurance bargaining causes providers to keep list prices of 5x what services actually cost. If you lack insurance, you will not only pay out of pocket, you'll pay 5x the price and go bankrupt.
  • No dental care for the poor. I estimate at least tens of thousands are left with decaying teeth they can't afford to fix. Most Americans can't afford a $500 surprise bill, and dental care is not covered by the ACA, so they don't have insurance. Tooth infections are left to deteriorate, with chronic pain that lasts for years, until the infection enters the bloodstream and the person visits the ER. Once there, it depends on the provider whether they will do what they can to save the person, or neglect to look at them seriously (so they die) because no one pays for it. Young people (with work, and young children!) have to rely on charity to not die due to their teeth.
Since this is unjust and causes pain, people talk about single-payer health care. This is a good idea, but it comes with problematic assumptions. A frequent one is that, under single-payer health care, the same money we're paying now will get everyone all the health care they need. This is not the case.

There are countries where single-payer systems appear to work, but have complaints. Complaints arise because no such system works by addressing all of their populations' health care needs. The potential cost of that would be infinite. It is in fact not the budget of the system that stretches to cover health care, it's the other way around. A budget is established, which is as much as people are willing to pay; then health care is made to fit within that budget.

When I say health care is "made to fit", I mean people are having to wait for health care in certain cases, and some of them die before they get it, or they can choose to pay for it themselves. This goes especially for the elderly. If you're old and need heart surgery, you get put on a queue that's 18 months long. If you survive by then, you get the surgery. If you do not, that's a saved cost for the system. If the queues are structured sensibly, this does not end up really affecting average life expectancy: people who end up not getting treatment are those who would die soon, anyway. However, it makes it appear as though the system is heartless, because it has to economize, and somehow ration access to health care.

If single-payer care existed in the US, it could not work any differently. Infinite amounts of care cannot be paid. If tax payers are willing to pay $4k per capita, that would be the budget, and the health care would be rationed out. This means not every obese diabetic necessarily gets treatment, and may die.

This differs from private health care, not in that care is not rationed, but that it is rationed so everyone gets something, instead of some people getting nothing, and others getting maximum possible care. Under single-payer, health care access is simply not completely based on the moneybags you have.

2016-08-17

Why software patents are counter-productive

I've said this before, but I need to keep saying it until the US patent system is reformed. This may mean forever, if necessary.

The existence of patents for software is not okay. There may be a so-so argument for patents where things cost $1 billion each to develop. Even there, publicly funded research could deliver solutions that focus on more real problems, and help more lives.

But there is no argument for things where the cost of development is cheaper than defending a patent, or even filing for one. For software, there is no reason to have patents. It's a pure intellectual land-grab.

Software that takes years to develop will not be founded on a single idea. It will rest on tens of thousands of ideas.

Each individual idea in a software product is low value on its own. The value is in building software that combines the ideas, and then maintaining that software to respond to users' needs. Most of the value of software development is in this process.

All software is protected by copyright. This is more so true for software that takes several years to write.

The presence of patents in this process does not act to preserve the crown jewel that's at the core of your software, because you do not have that type of crown jewel. Your crown jewel are your developers and your process.

If you now introduce patents to this process, all of the above is still the same, except now you can tie up your competitors in court on the basis of each of the ten thousand ideas used in their software, claiming that you own some form of right to that. In return, they must have a portfolio of patents themselves, so they can counter-claim that you violate their patents in your software.

This acts to remove small developers from the field, and divert resources from the work. The net benefit is negative.

It is generally most useful for ideas to be public. The main justification for the patent system is that it's a way for useful new ideas to become public, by way of granting their originators limited monopolies. This way they might not keep their best ideas secret, which is seen as a lesser evil than risking good ideas to be lost.

This system is only worth it for ideas that would not otherwise become public, or would not otherwise be discovered because discovery requires too much effort if not rewarded. The patent system is specifically not worth it where developing the idea requires less work than filing for a patent, or defending it.

This criterion is generally not met for any idea in software. Virtually all ideas that make software work are individually small and non-unique.

2016-07-30

Fear: An account of terror on top of a coal stack; by Franc Bider

[This story is by my dad, about his experience in the 1970s as a student in Kranj, Slovenia. What follows is my translation of the original Slovenian, performed July 29-30, 2016, blatantly without permission. The original must have been written, in its current telling, in the weeks before this. I added footnotes and the subtitle.]

If I try to describe it very precisely: I was squatting up there, all the way up there, on a pile of coal. With my head wrapped in spiderwebs that dangled from the sooty ceiling. In agony, I huddled into the dark corner, away, away from the meshed narrow window that shone its betraying light. My heart was pounding so hard I could hear it. It made an echo. And slowly sliding down the stairs to the cellar was Mrs. Fani...

I had never been so afraid in my life. Not before, nor after.

Oto proclaimed he would never live in that death chamber. By this, he meant the seven by ten feet of gross square footage, the worm-eaten nightstand by the tired bed, with that tall Franz Joseph-era altar for a headboard, and a wardrobe well known from black-and-white vampire movies. This comfort also comprised a green plastic wash basin, and four pints of water from the pitcher per day. And Mrs. Fani.

Neighbor Oto, who was to study vocational trade, gave up the tempting offer in a blink. And so, on these seventy square feet, my best student years unfolded.

Every morning, exactly at half before seven, Mrs. Fani knocked on the door and placed on the table a cup of hot malt coffee.1 Then, she journeyed to Mass on foot. On a shelf in front of the door, already waiting were all the relevant newspapers and magazines of that day. Each and every day in the villa under the Water Tower began that way.

I did not have a key to the front door. Nor a bathroom. I had a pitcher of water and the sound impression that I'm living in harmony with creation and the universe, inhabiting an old bourgeois villa infused with a ghost of some former culture and civility. Four of us intellectuals lived under the same roof: upstairs was Mr. Hubert, retired law man and professor of People's Self-Management, and on the ground floor were Mrs. Fani, her boisterous cat, and me.

Mr. Hubert spent evenings playing cards with friends, reading late into the night, and waking up at nine in the morning. For breakfast, he ate two bread rolls with ham and drank a white malt coffee. In his younger years, when his mamá still lived, he had already firmly embraced the wisdom that "beyond the second day, a guest shall not stay". In this, he gave no leeway. He would be especially consistent when it came to a female audience. He liked to travel, but he did not burden his bed in the second floor with a plural presence.

He set the curfew at ten thirty in the evening. When the last bus from Ljubljana arrives. Upsets at later times, he did not tolerate. When I was once ringing the bell at half past midnight, he went into such a racket that Mici the cat was not being herself for days.

Sometimes, rarely more so, he invited me into his heavenly kingdom. I remember relating to him my bicycle trip to Prague. I was scorning presumptuously the Czech standard of living at that time, their monuments to "the Russian army - our liberator". He disciplined me severely. That the standard of living is not just the number of different toothpastes at the grocery store. That the standard of living is also the number of schools, kindergartens, clinics and physicians that are available to the lower classes. So.

Whenever he thought that Mrs. Fani was offering me too detailed instructions about picking apples, cherries or plums, he benevolently scolded her: "But, Mrs. Fanči, Franček knooows..."

This became a folk saying which I later, when I was already deeply married, repeated several times myself: "But, Mrs. Fanči, Franček knooows..."

Mrs. Fani was from our parts. Of the Kódels from Negojnica. In Kranj, with the Korošec family, she had already served 40 years. Tirelessly, she slid up and down the stairs. In the kitchen, she kept house and friendship with Mici, the cat, who was a mighty curious cat and kept urging frivolously out of the kitchen into the world. So it was that Mrs. Fani had to, each and every way, unlock the kitchen door, reinsert the key on the other side, and again lock behind her. And in reverse, every time she returned.

As said, a rich forty years.

One time, Mr. Hubert asked me discretely if I would check about the wide Upper Savinja Valley, in case there seemed to be another, younger trustworthy lass, of which to keep a record. Because you never know... Of course, it soon turned out that the concern was unnecessary. As she had faithfully kept house for him, so Fanika buried her master, while he argued in his will granting her a lifetime right to stay in the old villa. The grieving others then slightly adjusted this lifetime right to their own desires and needs, handily chasing her out of the house within 14 days of the funeral (Ivan Cankar: The Servant Jernej and his cat Mica). Mrs. Fani continued to visit me occasionally for a number of years at the church house in Rečica, so that we reminisced together the idyllical life in Kranj. After forty years, the memories of Mr. Hubert were everything that was left of her life. And smooth cheeks. The face of her younger (married) sister was ornamentally interweaved with wrinkles, while the visage of Mrs. Fani remained tranquil and smooth all her life, like the face of Miša M.2

Ooh, yes...

Therefore, it could be suspected that visits in my room were not exactly desired. There wasn't space anywhere other than the bed, anyhow. Yet, it is understood that my future wife wanted to spend as much quality time with me as possible. We had both always decidedly strived to persuade one another into something.

While the stateroom was small, the window was just large enough so that the tiny creature could smuggle herself, without difficulty, into my student abundance.

As said, there was the most space in bed. Perhaps that's why we were mostly there. She had a number of good arguments, so discussions could draw late into the night. In those times, my arguments, too, were occasionally quite solid.

In the morning, at the arrival of coffee, she usually hid in the closet. Just till the danger left for church.

One day, though, the affair took an itty-bitty complication.

I don't know if the coffee was too hot, or her lips were too enticing, it seemed like chaplain Janez sped along with the Mass urgently - Mrs. Fani was, all at once, back at home. It happens.

Here, I must mention a few geographical facts: my room was on the ground floor, right by the kitchen and next to the stairs to the floor above us; a few stairs lower were the vestibule and a small room with the latrine; and there were also the stairs to the cellar.

When Mrs. Fani had gotten a hold of the newspapers and bread rolls and headed to the upper den, the two of us opened the door silently, ran slightly downstairs to the vestibule of the front door; the key was still in the lock.

But as if cursed - someone rang the bell just then. Mrs. Fani was already heading down the stairs, down, down, down...

My love swiftly hid in the water closet, I hesitated for a moment, missed the opportunity and stayed in the vestibule. I pretended as if I'd just come from the latrine, and slithered back into my room. Mrs. Fani unlocked the front door, took delivery of the home-raised eggs, and recompensed the neighbor. Mrs. Fani was tormented by occasional incontinence, but this time, thank god, she did not turn to the toilet. And she most likely also had to store the eggs to a cool place.

As soon as the lane was clear, I run to the water closet to set the frightened girl free. But in her haste, she had locked herself in, and now the key is stuck...

In a moment, Mrs. Fani is again out of the kitchen and in the foyer.

She asks, am I heading for classes?

Confused, I say y... yeaaa... yes.

She unlocks the door, waits for me to exit elegantly, then closes and locks behind me.

I'm locked outside, girl locked in the toilet!

When the cornered bird finally hears Mrs. Fani singing to her cat, she manages to unlock the latrine and hastily bustles her way out, out into the daylight of Kranj.

But dread, now the front door is unlocked. There is the dire hazard that the cat might get an inkling, crack open the unlocked door, and - goodbye! We'll be searching for her again over the entire city. There's no alternative: I have to get into the house! The door can't be unlocked!

I take a deep breath and return to the vestibule. Carefully and silently, I lock the door. And then, of course, into my room and out the window... Some other time!

The traffic is killer. Fanika is barging in again, from upstairs to ground floor. She's probably going to the kitchen. It would be nice if she went to the kitchen.

Of course she doesn't. She goes right past the kitchen, straight to the vestibule. Okay. So I sneak down into the basement.

Maybe she's gonna pee? Oh no! She's not gonna pee. She, too, is driven by some fatal attraction into the basement. Will she turn to the chickens? Will she grab pickled fruit? Will she feed the furnace?

As far as dear god can see, I have nowhere to hide. I climb on all fours up the coal. Further up. All the way up. I coat myself in spiderwebs. I'm squatting on top of the coals. Just under the ceiling arch. The cellar echoes the drumming of my heart. I'm dying of horror! Now, now, Mrs. Fani will turn the corner, grab the coal shovel, and get an eyeful of me hanging out there.

A few minutes ago she let me out of the house. She wished me farewell and god's blessings. She conscientiously locked the door behind -

And now he's squatting here in the cellar, on the coals, as pale as death!!

- How is it possible?

- What on Earth is he doing there?

- And he seemed like such a decent lad?

- They said he's from a respectable, pious family?

- We brought him coffee in bed... And on Sundays even Sunday lunch, because the eateries are closed, ostensibly -

???

And as if Mrs. Fani had sensed a dreadful, grave calamity, in that fateful moment she judiciously turned to her chickens.

Since then, I believe something exists above us.



1 Coffee substitute, common in Slovenia at this time. There's no real coffee in this story.

2 Miša Molk - a Slovenian TV presenter known for her eerily obvious cosmetic surgery.

2016-07-28

Code generation bug with new optimizer in Visual Studio 2015 Update 3

When I saw that Visual Studio 2015 Update 3 includes a brand new optimizer, my first thought was: yikes, I wonder how well this is tested. It can't be very mature code.

Well – it isn't. The following reproduces a code generation bug involving the optimizer and a lambda that captures local variables by reference:

 
  #include <iostream>
  #include <cstring>
  #include <conio.h>
  #include <windows.h>

  using namespace std;

  #define If(EXPR, T, A, B) \
      (([&]() -> T { if (EXPR) return (A); return (B); })())

  void Report(size_t x, size_t a, size_t b)
  {
      cout << x << endl;

           if (x == a && x > b) cout << "OK, result is strlen(argv[0])" << endl;
      else if (x == b && x > a) cout << "OK, result is strlen(argv[1])" << endl;
      else                      cout << "Error - result is garbage" << endl;
  }

  int main(int argc, char** argv)
  {
      if (argc < 2)
          cout << "Enter anything as first parameter" << endl
               << "If compiled with cl -EHsc, output will be "
                  "max(strlen(argv[0]), strlen(argv[1]))" << endl
               << "If compiled with cl -EHsc -O2 under VS 2015 Update 3 x64, "
                  "output will be a large garbage number" << endl
               << "If compiled with cl -EHsc -O2 under VS 2015 Update 3 x86, "
                  "output may or may not be a small garbage number" << endl;
      else
      {
          cout << "Opportunity to attach a debugger. "
                  "Press any key to continue" << endl;
          _getch();
          if (IsDebuggerPresent())
              DebugBreak();

          size_t a = strlen(argv[0]);
          size_t b = strlen(argv[1]);
          Report(If(a > b, size_t, a, b), a, b);
      }
      return 0;
  }

  /*

  Problem disassembly on x64:

  Optimizer appears to recognize that variables "a" and "b" never need to be
  stored on stack, so it gives them both a dummy stack location [rsp+40h].
  This location is never written to. But code generated for the If lambda
  returns values from the dummy stack location.

          size_t a = (size_t) strlen(argv[0]);
  00007FF60F43A41F  mov         rax,qword ptr [rbx]  
  00007FF60F43A422  or          r8,0FFFFFFFFFFFFFFFFh  
  00007FF60F43A426  mov         rdx,r8  
  00007FF60F43A429  nop         dword ptr [rax]  
  00007FF60F43A430  inc         rdx  
  00007FF60F43A433  cmp         byte ptr [rax+rdx],0  
  00007FF60F43A437  jne         main+0B0h (07FF60F43A430h)  
          size_t b = (size_t) strlen(argv[1]);
  00007FF60F43A439  mov         rax,qword ptr [rbx+8]  
  00007FF60F43A43D  nop         dword ptr [rax]  
  00007FF60F43A440  inc         r8  
  00007FF60F43A443  cmp         byte ptr [rax+r8],0  
  00007FF60F43A448  jne         main+0C0h (07FF60F43A440h)  
          Report(If(a > b, size_t, a, b), a, b);
  00007FF60F43A44A  cmp         rdx,r8  
  00007FF60F43A44D  lea         rax,[rsp+40h]  
  00007FF60F43A452  lea         rcx,[rsp+40h]  
  00007FF60F43A457  cmovbe      rcx,rax  
  00007FF60F43A45B  mov         rcx,qword ptr [rcx]  
  00007FF60F43A45E  call        Report (07FF60F4292F2h)  

  Example output:

  C:\Temp>VsUpdate3.exe a
  Opportunity to attach a debugger. Press any key to continue
  140694781775872
  Error - result is garbage

  */


Solution?

According to the MSDN blog post that introduces the new optimizer, it can be disabled using the undocumented flag:

  -d2SSAOptimizer-

The above snippet runs properly when compiled with this flag. The disassembly is:
  
        size_t a = strlen(argv[0]);
  00007FF7B48C9FAF  mov         rax,qword ptr [rbx]  
  00007FF7B48C9FB2  or          r8,0FFFFFFFFFFFFFFFFh  
  00007FF7B48C9FB6  mov         rdx,r8  
  00007FF7B48C9FB9  nop         dword ptr [rax]  
  00007FF7B48C9FC0  inc         rdx  
  00007FF7B48C9FC3  cmp         byte ptr [rax+rdx],0  
  00007FF7B48C9FC7  jne         main+0B0h (07FF7B48C9FC0h)  
        size_t b = strlen(argv[1]);
  00007FF7B48C9FC9  mov         rax,qword ptr [rbx+8]  
  00007FF7B48C9FCD  nop         dword ptr [rax]  
  00007FF7B48C9FD0  inc         r8  
  00007FF7B48C9FD3  cmp         byte ptr [rax+r8],0  
  00007FF7B48C9FD8  jne         main+0C0h (07FF7B48C9FD0h)  
        Report(If(a > b, size_t, a, b), a, b);
  00007FF7B48C9FDA  cmp         rdx,r8  
  00007FF7B48C9FDD  mov         rcx,r8  
  00007FF7B48C9FE0  cmova       rcx,rdx  
  00007FF7B48C9FE4  call        Report (07FF7B48B92F2h)  

Interestingly – the code looks cleaner and more optimized without the SSA Optimizer. :-)

Update blues

I first tried to work around this issue by downgrading back to Update 2. This is not for the faint-hearted:
  • After uninstalling Update 3, the development environment would no longer start; it would crash after displaying the splash screen.
  • The Microsoft pages that claim to offer the Update 2 installer perform a bait-and-switch, and link to the Update 3 installer instead.
  • I downloaded the full Update 2 ISO from MSDN Subscriber Downloads, only to find that it goes online, as well as consults locally cached data, and then also pushes Update 3.
This is after the Update 3 installer took 3 hours and 50 minutes to run, because it turns out it downloads 1.7 GB during the installation on-demand – and it does so much slower than it would have taken to download the full 6 GB ISO, all while preventing getting any work done due to installation in progress.

Fortunately, comments suggest the optimizer issue might be fixed within a week, so that this will no longer be an obstacle.

2016-05-22

The barren rockscape and the Garden of the Mind

Xara woke up one day, back when she didn't yet know she was a gardener. She thought her garden was a field, and she didn't much tend to it. So the field had some flowers in it – those that were hardy enough to grow; and it also had a lot of weeds. She liked to take walks in her field, enjoying the flowers she found. She ignored the ugly weeds, and cursed at the nasty ones that scratched her.

The weeds pestered her a bit, the nasty ones even more so. She thought that she should tend the field, from time to time, but the task seemed so much harder than the problem. She tried to pluck a weed here and there, but almost as soon as she did that, it seemed another one popped up. And the nasty ones, oh boy. They were easier to avoid.

This day though, she would find something different. In the corner of her field, she found a new flower. It was more beautiful than anything she'd seen, with vivid blossoms and mesmerizing leaves. She had seen plenty of decent enough flowers; but they were sturdy, ones that could grow among the weeds. This one... This one grew as if it was blessed with magic. Within a month, it had covered the entire field, and Xara enjoyed the view like she never did. But then... the vivid blossoms fell. The leaves withered, lost their color, and shrank into thorns. The incredible flower became a weed. A weed that now smothered the entire field it had overgrown, and made it impassable with thorns.

Xara cried, but she hadn't lost hope. She thought to herself, "Maybe it's a seasonal flower." So she waited. Seasons went by. She tried to enjoy the other flowers, which had slowly again started to grow. But nothing compared to the beauty of the mesmerizing flower that had turned her field into a thorn labyrinth. She waited in sadness, as the remains petrified. The magical flower would not grow.

She talked to her neighbors about the mystical flower. Some of them had also just noticed the flower in their fields, and had no idea why and how. Some of them had seen the flower also overgrow their fields and turn into weed, and were sad about it like she was. Some had never seen this flower, and thought she was insane when she described it. But none of them knew how to help.

There was word of a sage; a sage that lived on the outskirts, and knew about many things. Xara visited the sage, and explained what happened in her field.

The sage nodded. "Xara, your problem is clear."

"It is?" she wondered.

"You see, Xara, your problem comes from the topsoil. From where do the weeds grow?"

"Well, from the soil," Xara affirmed.

"Indeed. And where do the flowers grow?" he inquired.

"Well... also the soil."

"You see therefore, that as long as there is topsoil, there will be flowers, as well as weeds."

Xara thought. "Well... I guess... probably."

"Tell me. Have you ever dug under the soil?"

She would have liked him explain about the magical flower; but she did not dare ask. "I... have not."

"Then you have not seen what's under the soil. Do you want to know what's under it?"

"I... do," she replied timidly.

"Under the soil is bedrock." The sage paused, then proceeded ponderously. "Your field has these things happening in it – the small flowers, the small weeds; the magical flower, and now the encroaching weed – all because of the topsoil. You are attached to the flowers that grow from topsoil. But with flowers come also the weeds."

Xara began to see his point. The sage continued.

"Look how impermanent the flowers are. They grow, they blossom, they die. How can you find lasting joy in something so ephemeral?"

"Hmm," she pondered.

"Stillness is found in removing the soil. Then no flowers will grow, and neither will weeds. When you look out on your field, you will see the bedrock; permanent and unchangeable. This is beyond flower and weed. It is emptiness. It is stillness."

Xara thanked the sage for advice, and returned to her field. She pondered his words, and decided. She's going to remove the topsoil. There was a lot of it, so removing it took a long time.



At long last, the soil was gone from her field. The weeds and the flowers kept trying to grow as long as there was still something to grow in. But now the bedrock was exposed, as far as the eye could reach.

She knew that, even so, there was soil in the pockets and cracks. Plants were hardy, and could take root there. No matter; she removed them, she washed out the cracks, and the bedrock was bare. There were no flowers or weeds apart from this, but there was beauty in the rockscape. It had colorful gradients. It appeared still and permanent. Within timelessness, rain would fall. Puddles would form, and evaporate.

Xara forgot about the magical flower that had turned to weed. Hers was now a very different field.



Years later, a new neighbor moved in nearby. Xara paid him a visit.

"Howdy there, neighbor. How is your field?"

"It could be worse," the neighbor said. "It wasn't maintained when I got here. But I'm going to tend to it."

"You're going to tend to it?" Xara was puzzled. The very idea seemed silly. She remembered how ineffective it was, back when her field had topsoil, trying to remove the weeds. So much work, for so little reward. Now that she had bedrock, maintenance was easy.

"Oh, yeah," quipped the neighbor. "No problem at all. I have the tools."

Xara told him about removing topsoil from the field, and how much easier it made things. The neighbor nodded. He said, "Sure, some people prefer that. I suppose there is beauty in it." He didn't seem interested more, so she didn't push it.

A couple months later, Xara visited again. "Howdy, neighbor. How's your field?" She wondered if had given up on tending yet. Should she tell him again about removing topsoil from the field?

"Well, it's not a field any more! It's a garden. Do you want to come see?"

Xara was incredulous, but curiosity got the better of her. She followed the neighbor to his garden. It was incredible! Flowers everywhere; arranged in these beautiful shapes –

"This... is amazing," she said. "But don't you feel sad when they die?"

He chuckled. "It's natural for them to die. But then they grow again!"

She followed him on paths between flowers, beautifully arranged. And there - in the center - there was the mystical flower she'd seen, with the vivid blossoms she once wanted to see again so badly.

"This one..." She pointed, with her mouth agape.

"Ah yes, that's a beauty. Requires real special care. If I don't add nutrients to the soil every day, it uses what's in there, then overgrows everything looking for more, and when that runs out, turns to weed. Amazingly colorful though, isn't it? I have a sprinkler system here, for the enriched water it needs."



A few months from then, Xara had put the soil back in her garden. She learned that tending it is not as difficult as she thought it would be. It is different, certainly. There's lots to do, compared to the barren rockscape. But she learned from the neighbor to treat the work like it's fun, and just do it consistently. Day in and day out, her garden grew nicer and better, and she found great joy in this. She made room for all of the flowers. She found that some of the weeds are actually flowers too. The nasty ones especially! She put them off the path, so they wouldn't scratch. She enjoyed seeing them blossom in the spring.

And the mystical flower, it came back. Xara's smile was as wide as her face when she found it. And this time, she knew exactly how to tend to it.

2016-05-15

The benefit of some kind of authority

There are well-meaning people who like to question everything, who go around with ideas like:

"Hey, why should Government Guy be able to do things others can't? The very concept of government is oppression. Why do we need authority? Anarchy is freedom!"

Government exists (1) to serve as a mechanism of coordination, and (2) to prevent arising of a worse, more oppressive authority.

Authority, in general, is emergent. Wherever there isn't authority, one will arise. Not because people want to be oppressed, but because (1) coordination services are needed, and (2) there is no authority in place to prevent another from taking power with force.

In this sense, a situation without authority is in disequilibrium, whereas a situation with authority is in a local equilibrium.

Situations that are truly without authority are hard to maintain, and hard to get work done in. For an example, consider how long it took someone to invent Bitcoin, the first authority-less currency (or more accurately: a currency with truly competing authorities). It is currently an unsolved problem how to extend Bitcoin to process more than about 10 transactions per second without some type of centralization.

In comparison, Visa is a federated system that has no qualms relying on authority, and can process 20,000 transactions per second.

For another example, consider protocols for database cluster synchronization, like the Raft Consensus Algorithm. The link has a very nice visualization. One of the first things you might notice is that all the members of the database cluster are peers, they are equivalent to each other, but at any time they elect one of themselves as an authority to act as coordinator.

Most coordination problems are a lot easier to solve if you have authority. This is to the extent that you can have 20,000 transactions per second if you use authority, and 10 transactions per second without.

In this reality, authority is a natural consequence of the natural laws in place. It's up to us to design and maintain authorities that represent the best of our intentions.

That said, authorities are a liability – a single point of failure. When we can avoid authorities, we should. If Bitcoin survives the test of time – e.g. does not end up controlled by a cartel of 6 miners in China – I believe the internet Domain Name System would be best replaced with one based on Bitcoin. (I think not Namecoin, but a system backed by the full mining effort that goes into the Bitcoin blockchain.)

2016-04-20

On the desirable properties of a programming language

For the past 15 years, I've wanted to create a programming language. It is easy enough to do so, if one just wants to create something. But I've wanted to create something that improves in some substantial way on what already exists, and stands the test of time.

10 years ago, I created Flow. Flow is not widely known, but it's the asynchronous architecture that underpins Bitvise SSH Server and SSH Client. It's a message-passing architecture based on small, Erlang-like processes, implemented using fibers in C++. One of the main purposes of Flow was to make it easier to reason about a complex, asynchronous application, and I do believe it succeeded to an extent. Its grander motivation was to make it trivial to write complex applications that scale effortlessly to any number of parallel processors.

I believe that Flow would meet this second goal, but its ability to do so is yet untested. We do not actually need the type of scaling Flow would provide in our software. We use Flow inside traditional thread-based parallelization which works better with the architecture of our SSH Server. I have since written software such as a web server with an integrated database, and that doesn't need scaling on a component granularity either. While Flow has free scaling, it requires a shift in the programmer's perspective. Traditional thread-based approaches require no shift, and tend to work well enough.

I originally wanted to turn Flow into a language. I realize, now, that I could; but this language would have problems inherent in all languages that try to impose some way of thought on the programmer. By imposing this way of thought, the language alienates people to whom it is foreign. Secondly, the intentional restrictions of the language make it difficult to implement designs that people actually want to use. The language then has a choice: either condemn and not help with these patterns, so the language becomes obscure; or smuggle violations of its principal design through a backdoor.

And now, as prime example of backdoor smuggling of design violations, I present:

Haskell


Haskell is known as a pure functional programming language which is actually so well designed that it can run nearly as fast as C++. Or so we're told.

To achieve C-like speeds in Haskell, you have to brutalize the beautiful language by treating it, well, like C.

Exhibit one: Jan Stolarek's article where he presents the following algorithm that takes a list of machine-wide integers, and computes the sum of squares of odd numbers in it:
 
  sumSqrL :: [Int] -> Int
  sumSqrL = sum . map (^2) . filter odd

What a piece of beautiful code. But according to Jan's measurements, it performs 8 times slower than C. He fixes this by reimplementing the algorithm as follows:
 
  sumSqrPOp :: U.Vector Int -> Int
  sumSqrPOp vec = runST $ do
    let add a x = do
          let !(I# v#) = x
              odd#     = v# `andI#` 1#
          return $ a + I# (odd# *# v# *# v#)
    foldM` add 0 vec -- replace ` with ' here

This is not only practically unreadable; it is practically imperative. It looks like it would look in C++, except that C++ is actually designed for this, and doesn't make it look so ugly.

If this is not sufficiently convincing, consider the problem of efficient string concatenation in Haskell. Of course, Haskell can append to a string:
 
  xs `append` x = xs ++ [x]

The only problem is that, when repeated, this has quadratic performance. A string, you see, is represented as a list. So each time you append a single character, Haskell creates a new list; or if not that, at least traverses the whole existing list; and then appends the additional character.

I suggest exploring the Haskell solutions proposed in the above Stack Overflow link. Let me know if you think any of them are clear. In Haskell, string concatenation is an advanced topic.

Managed languages


The main shortcoming of managed languages; with prime examples .NET and Java; is that they require management infrastructure, which can't be written in the managed language. You can't write a garbage collector in a language that assumes there is a garbage collector, and in which you can't access memory.

The way I see it, managed languages are the wrong approach to solving the problem of reliability in software. Imagine there is a surgeon. As part of their job, the surgeon needs to cut people. The surgeon's hand is steady, but when cutting freehand with a scalpel, the surgeon sometimes makes mistakes.

To reduce the number of scalpel mistakes, we invent cutting templates. In the types of surgeries that are most common, the surgeon can now use a cutting template instead of using the scalpel freehand. This improves the safety of most surgeries, and is generally seen as a good thing.

C is like a surgeon without a cutting template. C++ is like a surgeon who has the choice of a cutting template, or a scalpel. Managed languages are like surgeons being treated like they're inept. You only get cutting templates; the scalpel is taken away. If the problem doesn't fit the templates – well – you can try to mangle it until it does.

Managed languages solve memory safety issues that were a problem in the 1990s and 00s, but are mostly gone in modern C++. What managed languages do not do is protect against incompetent programmers, who remain free to implement basic vulnerabilities to SQL injection and cross-site forgeries. Managed languages do not allow you to rely on programmers whom you couldn't trust with C++.

C++


So what's wrong with C++?

Not really that much. This:
 

  // Signed -> Signed

  template <class OutType, class InType,
    std::enable_if_t<(
      std::numeric_limits<InType >::is_signed &&
      std::numeric_limits<OutType>::is_signed &&
      sizeof(InType) > sizeof(OutType)
    ), int> = 0>
  inline OutType NumCast(InType value)
  {
    // Input type is larger
    EnsureThrow(value >= std::numeric_limits<OutType>::min());
    EnsureThrow(value <= std::numeric_limits<OutType>::max());
    return (OutType) value;
  }

  template <class OutType, class InType,
    std::enable_if_t<(
      std::numeric_limits<InType >::is_signed &&
      std::numeric_limits<OutType>::is_signed &&
      sizeof(InType) <= sizeof(OutType)
    ), int> = 0>
  inline OutType NumCast(InType value)
  {
    // Input type is NOT larger
    return (OutType) value;
  }

The above are two out of eight template functions that use SFINAE to implement NumCast - a mechanism we use to ensure at run-time that the value of one type of integer can be safely converted to another.

I show this to emphasize the clunkiness of C++ metaprogramming. The entire language is not so much designed, as it is undesigned. Each individual step in the language's evolution was carefully chosen. But from a bird's eye view, C++ ends up looking like a hack, built upon a hack, built upon a hack. These hacks mostly fit well together; there is actual beauty in their outcome, and satisfaction in using the language. But the above is not an example of intuitive metaprogramming.

Go? Rust?


I like Rust. Rust is smart. It has the obvious feature of rigid single-reference memory safety. I suspect this may be an "overarching idea" problem, and may require smuggling of design violations by the back door.

"Go" seems like a language you design if you want everything simple, and have high developer turnover. It's a systems language for when your developers are from JavaScript, and don't want to learn C++.

Where to go?


The kind of language I think is worthy of creating:
  • Can express any type of program. Not just a garbage collected program, or just a user-mode program; but also a kernel-mode program, a boot loader, or an embedded real-time program.
  • Does not impose limiting paradigms which then need to be broken through a backdoor. No functional Fata Morgana, no object-oriented Fata Morgana, no message-passing Fata Morgana. The language should allow these concepts to be expressed, but not foist them.
  • Has regular structure and strong built-in metaprogramming. This is a lesson that can be learned from LISP (hint hint hint). The main problem with LISP is all the parentheses.
It might not surprise that I have in mind a design for just such a language. :) In a future post, I would like to introduce Seed – a list-based, self-building front-end language, with a simple but powerful basic structure, from which a versatile platform can be built for any purpose. It will first be intended to target the LLVM intermediate representation; but there's no reason it could not also emit .NET or Java bytecode, platform-specific machine code, HTML, PDF, or text files.

Seed is also intended to not have subtly incompatible implementations, which continue to be an issue with C++. The idea is for most of the language to be defined in the language itself. Implementations are then much more likely to be compatible, as they use the same language definition.

And if you don't like the language? Change it. Most of the language will be defined in the language itself, so its definition is under your control.

2016-04-16

"Hello world" in LLVM IR language without CRT

The LLVM Language Reference Manual has a "Hello world" example that uses the C run-time library.

I wanted a bare-bones example, using only the Win32 API, without a CRT dependency. Here it is:
  @.str = private unnamed_addr constant [13 x i8] c"Hello world\0A\00"

  declare i64 @GetStdHandle(i32) nounwind
  declare i32 @WriteFile(i64, i8* nocapture, i32, i32* nocapture, i8*) nounwind

  define i32 @Entry() {
    %hwStr = getelementptr [13 x i8]* @.str, i64 0, i64 0
    %handle = call i64 @GetStdHandle(i32 -11)
    %bytes = alloca i32
    call i32 @WriteFile(i64 %handle, i8* %hwStr, i32 12, i32* %bytes, i8* null)
    ret i32 2
  }
Compile and link steps:
  llc -filetype=obj hw.ll
  link hw.obj kernel32.lib /entry:Entry /subsystem:console
The resulting executable has 2,560 bytes, prints "Hello world", and returns exit code 2.

My main observation is that the language is clumsy. It's meant to be read by humans, but in most cases, generated automatically:
  1. No syntax for hexadecimal integers. If you want to enter a value like 0xC0000005 – good luck! :-)
  2. The types of all values have to be explicitly qualified each time they're used. This is irritating when writing, but may serve as useful documentation when reading.
  3. Interestingly, there's no "address-of" operator. Instead, alloca creates space for a value on the stack, and gives you the address. Neat!