Youry's Blog

Youry's Blog

Archive for the ‘IT’ Category

Ubuntu16/Windows2012/CentOS7 cheat-sheet (started 23 May 2016)

leave a comment »

CentOS7

  1. VMWare tools prereq: yum install perl gcc make kernel-headers kernel-devel -y
    http://www.ehowstuff.com/how-to-install-vmware-tools/
  2. aborted yum transactions should be cleaned up: yum-complete-transaction –cleanup-only
    http://forums.fedoraforum.org/showthread.php?t=270035
    http://idroot.net/tutorials/fix-yum-duplicate-packages-broken-dependencies/
    http://forums.fedoraforum.org/showthread.php?t=194254
  3. How to reboot headless server with passphrase? here there are 2 of the best ones I’ve managed to found so far (after spending lots of hours trying out multiple impossible configurations ¬¬):

    https://stinkyparkia.wordpress.com/2014/10/14/remote-unlocking-luks-encrypted-lvm-using-dropbear-ssh-in-ubuntu-server-14-04-1-with-static-ipst/

    https://github.com/chadoe/luks-triple-unlock/blob/master/install.sh

    After configuring the servers that way, one can decrypt the LUKS partition via SSH (using password or rsa-key) or an USB flash drive and let the system boot as usual. Quite useful in my case.

 

Ubuntu 16_04 64bit

  1. Static IP is not updated. Must restart server.

Windows 2012 R2 DC

  1. Firewall must be updated explicitly for the 3389 port to get remote desktop work.

Written by youryblog

May 23, 2016 at 10:11 PM

Posted in IT, UNIX/Linux tricks

Useful: Move Your Apple Mail to a New Mac or a Clean Install Of OS X

leave a comment »

From http://macs.about.com/od/usingyourmac/ss/moving-apple-mail.htm#showall

Published: 11/10/2009

Updated: 1/19/2015

Worked for me, but didn’t get accounts in System Preferences. Had to add email accounts manually to get calendars, but email accounts was not updated. As a temporary solution this is good enough.

Move Your Apple Mail to a New Mac or a Clean Install Of OS X

Setting up Mail again from scratch is a waste of time. Instead, migrate your Mail from a previous Mac.

Moving your Apple Mail to a new Mac, or to a new, clean install of the OS, may seem like a difficult task but it actually only requires saving three items and moving them to the new destination.

There are a few ways to perform the move. By far the easiest, and the most often suggested method is to use Apple’s Migration Assistant. This method works well in most cases, but there’s one drawback to the Migration Assistant.

Draw flowcharts & collaborate with others online. Free 14-day trial!
Clean Mac files Right Now. Award-winning System Utility!
Backup your computer today Easy to use,Secure and Safe storage

Its approach is mostly all-or-nothing when it comes to moving data. You can select some basic categories, such as applications or user data, or just support files, and most of the time it works fine.

Where you can run into problems is when there’s something wrong with your Mac. You’re not sure what it is, maybe a corrupt preference file or a system component that’s a little whacky, and causes problems now and then. The last thing you want to do is copy a bad file to your new Mac or new install of OS X. But starting over completely doesn’t make sense, either. You may have years of data stored on your Mac. While some of it may be fluff, other pieces of information are important enough to keep on hand.

While it may be easy to recreate your mail accounts on a new system, it’s not easy to start off fresh, with none of your older email available, your Mail rules gone, and Mail always asking for passwords that you may have long since forgotten.

With that in mind, here’s a simple way to move just the data Apple Mail needs to a new location.

When you’re done, you should be able to fire up Mail on your new system and have all your emails, accounts, and rules working just the way they did before the move.

What You Need to Move Mail

  • A way to transfer files to the new location. You can transfer your files over a network, burn them to a CD or DVD, copy them to a USB flash drive, or, if the new system is on the same Mac, copy them from one hard drive partition to another. We won’t discuss the actual mechanism you use to perform the transfer, only which source files need to be copied, and where they need to be stored in your new installation.
  • Administrative access to your data. You may need to change the file privileges, although for most users, this will probably not be necessary.

If you’re all set, then let’s get started.

 

of 6

Back Up Before Moving Mail Data

If you’re using Time Machine for backups, be sure to add the Time Machine status item to your menu bar.

Before you start moving files around, make sure you have a current backup of your mail.

Back Up Data Using Time Machine

Select the ‘Back Up Now’ item from the ‘Time Machine’ icon in the menu bar or right-click the ‘Time Machine’ icon in the Dock and select ‘Back Up Now’ from the pop-up menu. If you don’t have the Time Machine menu bar item, you can install it by doing the following:

  1. Launch System Preferences by clicking the ‘System Preferences’ icon in the Dock, or selecting ‘System Preferences’ from the Apple menu.
  1. Select the ‘Time Machine’ preference pane in the System Preferences window.
  2. Place a check mark next to ‘Show Time Machine status in the menu bar.’
  3. Close System Preferences.

You can also create a backup using one of many third-party applications. Once you back up your data, you’re ready to continue.

of 6

When Moving Apple Mail Copy Your Keychain Data

Before copying your Keychain files, make sure they are in good shape by running Keychain First Aid.

There are two folders and a file that need to be copied to your new Mac or your new system. You will actually be copying data for both Apple Mail and Apple’s Keychain application. The Keychain data you copy will allow Apple Mail to operate without asking you to supply all of your account passwords. If you only have one or two accounts in Mail, then you can probably skip this step, but if you have many Mail accounts, this will make using the new Mac or system easier.

Before you copy the Keychain files, it’s a good idea to repair the files to ensure the data within them is intact.

Repair Your Keychain Files

  1. Launch Keychain Access, located in /Applications/Utilities.
  2. Select ‘Keychain First Aid’ from the Keychain Access menu.
  3. Enter the User Name and Password for the user account you are currently logged in with.
  4. You can perform just a ‘Verify’ to see if anything is wrong, or you can select the ‘Repair’ option to verify the data and repair any problems. Since you have already backed up your data (you did back up your data, right?), select ‘Repair’ and click the ‘Start’ button.
  5. When the process is complete, close the Keychain First Aid window, and then quit Keychain Access.

Copy the Keychain Files to the New Location

OS X Lion and later, hides the users Library. You can make the username/Library folder visible by following the guide OS X Lion Is Hiding Your Library Folder.

  1. Open a Finder window by clicking the ‘Finder’ icon in the Dock.
  2. Navigate to username/Library/, where ‘username’ is the name of your home directory.
  1. Copy the Keychain folder to the same location on your new Mac or in your new system.

 

of 6

Copying Your Apple Mail Folder and Preferences To a New Mac

The com.apple.mail.plist file contains your Mail accounts and preferences.

Moving your Apple Mail data is a pretty simple task, but before you do, you may want to take a bit of time to clean up your current Mail setup.

Apple Mail Cleanup

  1. Launch Apple Mail by clicking the ‘Mail’ icon in the Dock.
  2. Click the ‘Junk’ icon, and verify that all of the messages in the Junk folder are indeed junk messages.
  3. Right-click the ‘Junk’ icon and select ‘Erase Junk Mail’ from the pop-up menu.

Apple Mail Rebuild

Rebuilding your mailboxes forces Mail to re-index each message and update the message list to accurately reflect the messages actually stored on your Mac. The message index and the actual messages can sometimes get out of sync, usually as the result of a Mail crash or an unintended shutdown. The rebuild process will correct any underlying issues with your mailboxes.

If you use IMAP (Internet Message Access Protocol), the rebuild process will delete any locally cached messages and attachments, and then download fresh copies from the mail server. This can take quite a while; you may decide to forgo the rebuild process for IMAP accounts.

  1. Select a mailbox by clicking once on its icon.
  2. Select ‘Rebuild’ from the Mailbox menu.
  3. Once the rebuild is done, repeat the process for each mailbox.

Copy Your Mail Files

OS X Lion and later, hides the users Library. You can make the username/Library folder visible by following the guide OS X Lion Is Hiding Your Library Folder.

  1. Quit Apple Mail if the application is running.
  1. Open a Finder window by clicking the ‘Finder’ icon in the Dock.
  2. Navigate to username/Library/, where ‘username’ is the name of your home directory.
  3. Copy the Mail folder to the same location on your new Mac or in your new system.

Copy Your Mail Preferences

  1. Quit Apple Mail if the application is running.
  2. Open a Finder window by clicking the ‘Finder’ icon in the Dock.
  3. Navigate to username/Library/Preferences, where ‘username’ is the name of your home directory.
  4. Copy the ‘com.apple.mail.plist’ file to the same location on your new Mac or in your new system.

That’s it. With all the necessary files copied to the new Mac or system, you should be able to launch Apple Mail and have all of your emails in place, your Mail rules functioning, and all Mail accounts working.

 

of 6

Moving Apple Mail – Troubleshooting Keychain Issues

The Keychain list shows which Keychains are being shared. If a Keychain is shared, you may not be able to replace it.

If something can go wrong, it usually will, and moving Keychains around can cause a problem. Luckily, it is easy to correct.

Problems With Keychain

When you try to copy the Keychain file to its new location on your new Mac or system, the copy may fail with a warning that one or more Keychain files is in use. This can happen if you have already used your new Mac or system, and in the process, it created its own Keychain files.

To work around the problem, try the following:

  1. Launch Keychain Access, located in /Applications/Utilities, on your new Mac or system.
  2. Select ‘Keychain List’ from the Edit menu.
  3. Make a note of which Keychain files in the list have a check mark next to their name.
  4. Uncheck any checked Keychain files.
  5. Repeat the instructions on Page 3 to copy the Keychain files to your new Mac or system.
  6. Reset the check marks in the Keychain list to the state you noted above.

 

of 6

Moving Apple Mail – Troubleshooting Mail Issues

Your Mail files should list you as having ‘Read & Write’ privileges.

Moving mail files between systems can cause permission problems. Fortunately, these problems are easy to correct.

Problems With Copying Mail Files

Occasionally, you may run into a problem when you first launch Apple Mail on your new Mac or system. The error message will usually tell you that Mail does not have permission to access a file. The usual culprit is username/Library/Mail/Envelope Index. Make a note of which file is listed in the error message, then do the following.

  1. Quit Apple Mail, if it’s running.
  2. Open a Finder window by clicking the ‘Finder’ icon in the Dock.
  3. Navigate to the file mentioned in the error message.
  4. Right-click the file in the Finder window and select ‘Get Info’ from the pop-up menu.
  5. In the Get Info window, expand the ‘Sharing & Permissions’ item.
  6. Your username should be listed as having Read & Write access. You may find that, because the account IDs between your old Mac and the new system are different, instead of seeing your username listed, you see ‘unknown.’ To change the permissions, do the following:
  7. Click the lock icon in the bottom right corner of the Get Info window.
  8. Enter your administrator username and password, and click ‘OK.’
  9. Click the ‘+’ (plus) button.
  10. The ‘Select a New User or Group’ window will open.
  11. From the list of users, click your account, and click ‘Select.’
  12. The selected account will be added to the Sharing & Permissions section.
  13. Select the ‘Privileges’ item for the account you added in the Get Info window.
  14. From the Privileges dropdown menu, select ‘Read & Write.’
  1. If there is an entry with the name ‘unknown,’ select it, and click the ‘-’ (minus) sign to delete the entry.
  2. Close the Get Info window.

That should correct the problem. If Apple Mail reports a similar error with another file, you may want to just add your username to every file in the Mail folder using the Propagate command.

Propagating Your Privileges

  1. Right-click the Mail folder, located at username/Library/.
  2. Using the instructions above, add your username to the Permissions list, and set your permissions to ‘Read & Write.’
  3. Click the gear icon at the bottom of the Get Info window.
  4. Select ‘Apply to enclosed items.’
  5. Close the Get Info window and try launching Apple Mail again.

You can also try reseting user permissions, if all else fails.

That’s it. You should be ready to go with Apple Mail.

Published: 11/10/2009

Updated: 1/19/2015

Written by youryblog

July 11, 2015 at 9:07 PM

Posted in MacOS

Home Row Computing on Macs from http://duartes.org/gustavo/blog/

leave a comment »

It can be interested for the mac lovers:

It can be interested for the mac lovers:

Home Row Computing on Macs

http://duartes.org/gustavo/blog/ Nov 24th, 2014

For a number of years I’ve configured my desktops so that most tasks can be done using only home row keys on the keyboard, a technique I call home row computing. It takes the Vi idea of staying on the home row to every app, all the time, but without using modes so things are simpler.

I’ve described an implementation for Windows, but I have since moved to Macs and back to a qwerty keyboard (away from Dvorak). The current setup is described in this post. It uses familiar Vi key bindings and is far more suitable. It’s fairly painless to configure on the Mac and has never given me any problems, thanks to Takayama Fumihiko’s awesome keyboard apps.

Using this is a joy. It’s really fast, easy on the hands, and makes you feel like a geek god. If you don’t use Vim, you’ll now have one of its benefits in your favorite editor and in other apps, plus a weapon against smug Vimmers. If you already use Vim, your cherished hjkl keys become universal and pressing Esc gets a hell of a lot easier.

Some of the important keys that must be moved to home row are the arrow keys, Esc, delete (backspace) and forward delete. Another helpful home row task is moving and resizing windows. The key to all this is remapping Caps Lock to allow combinations of Caps Lock plus a home key to do these tasks. Again, there are no modes involved here, Caps Lock works as a modifier like the cmd and fn keys. Here’s a good start:

I have left several keys unmapped so you can customize your own setup, and we’ll get to window management in a moment. The first step is to set Caps Lock to No Action in System Preferences > Keyboard > Modifier keys:

Now we must remap the Caps Lock key code to something else. To do so, you need a small tool called Seil (open source). You can map Caps Lock to any other key, like cmd or option. So if you don’t want to go all-out home row, you can still benefit from the remapping.

I like to remap Caps Lock into something that guarantees no conflicts ever for our combos. So I use key code 110, which is the Apps key on a Windows keyboard and is safely absent from Apple keyboards:

Now we’re in business, the world – or at least the keyboard – is our oyster. The maker of Seil also makes Karabiner, open as well and an outstanding keyboard customizer for OS X. I have no affiliation with these tools, apart from being a happy user for years. If you end up using them, please donate. So go ahead and install Karabiner, and you’ll see a plethora of keyboard tweak possibilities:

Each of the tweaks can be toggled on and off. There are even native Vi, Vim, and Emacs modes. However, I don’t like the built-in ones, so I built my own config. Go to Misc & Uninstall and click Open private.xml:

In this file, ~/Library/Application Support/Karabiner/private.xml, you can define your own keyboard remapping scheme. I actually symlink that to a Dropbox file to keep the configuration consistent across my machines, but at any rate, here is a file you can use to implement what we have discussed so far. Drop the file in, click ReloadXML and you’ll have this:

Home Row Computing is at the top (prefixed with ! for sorting). Toggle it on, and you’re done. Enjoy your new keyboard layout, do a search on Spotlight and see how fast and smooth it is to choose an option.

Finally, there is window management. That’s an area where you can fumble quite a bit, resizing and moving about clumsily with a mouse. My favorite options to make it fast and homerow-friendly are ShiftIt (open) and Moom (best $10 I ever spent, no affiliation). There are some others, but to me Moom towers above the rest. It has a great two-step usage, where one hot key activates it:

And the following key triggers a command you get to define using window primitives like move, zoom, resize, and change monitors. You can also define shortcuts that run commands directly. Moom has some handy default actions:

Out of box, arrow keys can be used to send a window to the left, right, top, or bottom of the screen, and Moom natively interprets hjkl as arrows making it easy to stay on home row. You can associate keys with various commands and precise window positions:

This is gold for large monitors like Apple Thunderbolts. I remap Caps Lock + M into the global Moom shortcut for painless activation. This allows me to set the shortcut itself to something bizarre that won’t conflict with anything but would be a dog to type. Currently it’s an improbable Fn + Control + Command + M. I also have Caps Lock + N activating a Moom command that cycles a window between my two monitors. Both of these shortcuts are in the keyboard map I provided.

If you have any questions, let me know. I know a number of keyboard nuts out there use this scheme on Windows and Linux, and I hope this makes it easy to do so on Macs.

Written by youryblog

May 15, 2015 at 6:00 PM

Posted in MacOS

Linux / Fedora / CentOS Raid issues

leave a comment »

  1. http://blag.felixhummel.de/admin/raid.html
    how to create raid on large disks:

    gdisk /dev/sdc
    mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1

Written by youryblog

February 9, 2015 at 7:29 PM

Posted in UNIX/Linux tricks

Yosemite, iOS 8, Spotlight, and Privacy: What you need to know By Rene Ritchie, Monday, Oct 20, 2014 a 8:31 pm EDT

leave a comment »

According to Landon Fuller, who collected the data in the first place,
this is not just about Spotlight, and the data will continue to be
sent to Apple even if Spotlight Suggestions -- or any of a number of
other seemingly relevant system configuration options -- are disabled.

See

https://github.com/fix-macosx/yosemite-phone-home

for the raw data and analysis, without either the Apple apologism of
iMore or the journalistic spin of the Washington Post article they
cite.

Of course it is in Apple's interest to say that they care about
security and privacy, to emphasize how much effort they put into
minimizing data (we've heard this one from James Clapper before!), and
to claim that their snooping serves to benefit users by providing more
accurate answers.  None of this changes the surveillance they have
built into their system or how difficult it is to avoid!

Yosemite, iOS 8, Spotlight, and Privacy: What you need to know
By Rene Ritchie, Monday, Oct 20, 2014 a 8:31 pm EDT
http://www.imore.com/yosemite-ios-8-spotlight-and-privacy-what-you-need-know

A story made the rounds earlier today calling into question the new Spotlight Suggestions feature in OS X Yosemite and iOS 8. In an effort to garner attention, it reports the collection and usage of the information required to enable this feature in a needlessly scary way. As any long time reader knows, security and privacy are always at odds with convenience, yet features like Spotlight Suggestions — and Siri before it — do an excellent job balancing as much convenience as possible with maintaining as much privacy and security as possible. Here’s Apple’s statement on the matter:

“We are absolutely committed to protecting our users’ privacy and have built privacy right into our products,” Apple told iMore. “For Spotlight Suggestions we minimize the amount of information sent to Apple. Apple doesn’t retain IP addresses from users’ devices. Spotlight blurs the location on the device so it never sends an exact location to Apple. Spotlight doesn’t use a persistent identifier, so a user’s search history can’t be created by Apple or anyone else. Apple devices only use a temporary anonymous session ID for a 15-minute period before the ID is discarded.

“We also worked closely with Microsoft to protect our users’ privacy. Apple forwards only commonly searched terms and only city-level location information to Bing. Microsoft does not store search queries or receive users’ IP addresses.

“You can also easily opt out of Spotlight Suggestions, Bing or Location Services for Spotlight.”

Here’s the original charge:

Apple has begun automatically collecting the locations of users and the queries they type when searching for files with the newest Mac operating system, a function that has provoked backlash for a company that portrays itself as a leader on privacy.

The “backlash” cited by the sensationalistic story is not the result of the story but the result of sensationalism, and that’s disappointing. We depend on major publications to provide us with accurate information for our benefit, not for their own benefit. Where they could have taken the time to look into it, assess the facts, and help people understand, they chose to double down on FUD, and that’s not only disappointing, it’s distressing.

So what are the facts? Apple discloses how Spotlight Suggestions work in both the Spotlight section of System Preferences on the Mac, and in the Spotlight section of Settings > General on iPhones and iPads.

There’s also a Spotlight Suggestion check box on both so that you, the person using the device, can easily turn it off if you value privacy and security over convenience. (And if you are such a person, and have already disabled location services, Spotlight honors that setting and doesn’t send the information.)

Apple links to the following text right from the prefs/settings pane on both OS X and iOS. Not only is it simple to find, it’s plainly written and understandable:

When you use Spotlight, your search queries, the Spotlight Suggestions you select, and related usage data will be sent to Apple. Search results found on your Mac will not be sent. If you have Location Services on your Mac turned on, when you make a search query to Spotlight the location of your Mac at that time will be sent to Apple. Searches for common words and phrases will be forwarded from Apple to Microsoft’s Bing search engine. These searches are not stored by Microsoft. Location, search queries, and usage information sent to Apple will be used by Apple only to make Spotlight Suggestions more relevant and to improve other Apple products and services.

If you do not want your Spotlight search queries and Spotlight Suggestions usage data sent to Apple, you can turn off Spotlight Suggestions. Simply deselect the checkboxes for both Spotlight Suggestions and Bing Web Searches in the Search Results tab in the Spotlight preference pane found within System Preferences on your Mac. If you turn off Spotlight Suggestions and Bing Web Searches, Spotlight will search the contents of only your Mac.

You can turn off Location Services for Spotlight Suggestions in the Privacy pane of System Preferences on your Mac by clicking on “Details” next to System Services and then deselecting “Spotlight Suggestions”. If you turn off Location Services on your Mac, your precise location will not be sent to Apple. To deliver relevant search suggestions, Apple may use the IP address of your Internet connection to approximate your location by matching it to a geographic region.

Apple has also posted a privacy section on their website, and an updated version of their iOS 8 security document that reiterate what they’re doing and their long-standing position on privacy. Here’s the relevant parts:

To make suggestions more relevant to users, Spotlight Suggestions includes user context and search feedback with search query requests sent to Apple.

Context sent with search requests provides Apple with: i) the device’s approximate location; ii) the device type (e.g., Mac, iPhone, iPad, or iPod); iii) the client app, which is either Spotlight or Safari; iv) the device’s default language and region settings; v) the three most recently used apps on the device; and vi) an anonymous session ID. All communication with the server is encrypted via HTTPS.

The white paper goes on to explain how locations are blurred, anonymous IDs are only kept for 15 minutes, recent apps are only included if they’re on a white list of popular apps, etc. (It starts on page 40 of the above-linked PDF if you’re curious about the specifics.)

So, again, Apple is only doing what they need to do to provide the conveniences of the feature they announced — the same way they’ve needed to collect enough data to answer questions with Siri in the past, or show you locations on Maps, or find your iPhone, iPad or Mac, and the list goes on.

If you don’t like or want it, you can turn it off. That’s the real story here — education. How it works, and what you can do with it and about it.

If you have any concerns or questions about Spotlight Suggestions, let me know in the comments!

Written by youryblog

October 24, 2014 at 2:17 PM

Bluerays

leave a comment »

A disc with a capacity of 25 gigabytes is equivalent to 23.28 gibibytes, depending on whether you are using a base-2 system or a base-10 system. Additionally, some of that space is reserved as overhead for the file system as well as error-correction information. This results in a true recordable capacity of 23.28 gibibytes instead of the 25GB stated on the package. http://blog.cdrom2go.com/2010/12/maximum-capacity-of-a-blu-ray-disc/

The same situation is true of dual-layer Blu-ray discs. These are normally rated with a 50GB capacity, but in practice you can record about 46.57GB of data on a disc. Again, this is due to differences in the way the disc capacity is measured plus a small percentage of space reserved for file system information.

Written by youryblog

October 6, 2014 at 1:07 AM

Posted in HW, IT, MacOS

Programming without objects 17.09.2014 Permalink

leave a comment »

a copy from http://www.falkoriemenschneider.de/a__2014-09-17__Programming-without-objects.html

Programming without objects

17.09.2014 PermalinkRecently I gave an itemis internal talk about basic functional programming (FP) concepts. Towards the end I made a claim that objects and their blueprints (a.k.a. classes) have severe downsides and that it is much better to “leave data alone”, which is exactly how Clojure and Haskell treat this subject.

I fully acknowledge that such a statement is disturbing to hear, especially if your professional thinking was shaped by OOP paradigm1 for 20 years or more. I know this well, because that was exactly my situation two years ago.

So let’s take a closer look at the OO class as provided by Java, Scala or C# and think a little bit about its features, strengths and weaknesses. Subsequently, I’ll explain how to do better without classes and objects.

The OO class

An OO class bundles data and implementation related to this data. A good OO class hides any details about implementation and data, if they’re not supposed to be part of the API. According to widespread belief, the good OO class reduces complexity because any client using instances of it (a.k.a the objects) will not and cannot deal with those implementation details.

So, we can see three striking aspects here: API as formed by the public members, data declaration and method implementation. But there is much more to it:

  • Instances of a class have identity by default. Two objects are the same if and only if both share the same address in memory which gives us a relation of intensional equality on their data.
  • The reference to an object is passed as implicit first argument to any method invocation on the object, available through this or self.
  • A class is a type.
  • Inheritance between classes enables subtyping and allows subclasses to reuse method implementations and data declarations of their superclasses.
  • Methods can be “polymorphic” which only means that in order to invoke a method objects do a dispatch over the type of the implicit first argument.
  • Classes can have type variables to generalize their functionality over a range of other types.
  • Classes act as namespaces for inner classes and static members.
  • Classes can support already existing contracts of abstract types (in Java called “interfaces”) by implementing those.

That’s a lot of stuff, and I probably did not list everything that classes provide. The first thing we can state is that a class is not “simple” in the sense that it does only one thing. In fact, it does a lot of things. But so what? After all, the concept is so old and so widely known that we may accept it anyway.

Besides the class being a complex thing there are other disadvantages that most OO programmers are somehow aware of but tend to accept like laws of nature2.

Next I’ll discuss a few of these weaknesses in detail:

Equality

Let’s look at intensional equality as provided by object identity. Apparently this is not what we want at all times, otherwise we wouldn’t see this pattern called ValueObject 3. A type like Javas String is a well-known example of a successful ValueObject implementation. The price for this freedom of choice is a rule that the == operator is almost useless and we must use equals because only its implementation can decide what kind of equality relation is in effect. The numerous tutorials and blog posts how to correctly implement equals and hashCode are witness that this topic is not easy to get right. In addition we must be very careful when using a potentially mutable object as a key in a map or item in a set. If it’s not a value a lookup might fail miserably (and unexpectedly).

Concurrency

The conflation of mutable data and object identity is also a problem in another regard: concurrency. Those objects not acting as values must be protected from race conditions and stale memory, either by thread confinement or by synchronization which is hard to do right in order to avoid dead locks or live locks.

This

Let’s turn to the implicit first argument, in Java and C# it’s referred to by the keyword this. It allows us to invoke methods on an object Sam by stating Sam.doThis() or Sam.doThat() which sounds similar to natural language. So, we can consider this as syntactic sugar to ease understanding in cases where a “subject verb” formulation makes sense4. But wait, what happens if we’re not able to use a method implemented in a class? More recent languages offer extension methods (in Scala realizable using implicit) to help make our notation more consistent. Thus, in order to put the first argument in front of the method name to improve readability we’re required to master an additional concept.

Dispatch

Objects support polymorphism, which was the really exotic feature back then when I learned OOP. Actually, the OO “polymorphic method invocation” is a special case of a function invocation dispatch, namely a dispatch over the type of the implicit first argument. So this fancy p-thing turns out to be a somewhat arbitrary limitation of a very powerful concept.

Inheritance

Implementation inheritance creates a “is-a” relation between types (giving birth to terms like “superclass” and “subclass”), shares API, and hard-wires this with reuse of data declarations and method implementations. But what can you do if you want one without the other? Well, if you don’t want to reuse but only define type relations and share API you can employ interfaces, which is simple and effective. Now the other way around: what can you do to reuse implementations without tying a class by extends to an existing class? The options I know are:

  • Use static methods that expect a type declared by an interface (which might force some data members to become publicly accessible). Here again, extension methods might jump in to reduce syntatic disturbances.
  • Use delegation, so the class in need for existing implementations points to a class containing the shared methods. Each delegating method invokes the implementation from the other class, which creates much boilerplate if the number of methods is high.

Implementation inheritance is a very good example for adding much conceptual complexity without giving us the full power of the ideas it builds upon. In fact, the debate about the potential harm that it brings (and how other approaches might remedy this) is nearly as old as the idea itself.

Encapsulation of data

“Information hiding” makes objects opaque, that’s its purpose. It seems useful if objects contain private mutable data necessary to implement stateful behaviour. However, with respect to concurrency and more algebraic data transformation this encapsulation hurts. It hurts because we don’t know what’s underneath an objects API and the last thing we want to have is mutable state that we’re not aware of. It hurts that if we want to work with its data we have to ask for every single bit of it.

The common solution to the latter problem is to use reflection, and that’s exactly what all advanced frameworks do to uniformly implement their behaviour on top of user-defined classes that contain data.

On the other hand, we all have learned that reflection is dangerous, it circumvents the type system, which may lead to all kinds of errors showing up only at runtime. So most programmers resort to work with objects in a very tedious way, creating many times more code compared to what they would have written in a different language.

Judging the OO class

So far, we have seen that OO classes are not ideal. They mix up a lot of features while often providing only a narrow subset of what a more general concept is able to do for us. Resulting programs are not only hard to do right, in terms of lines of code they tend to require a multiple compared to programs written in modern FP languages. Since system size is a significant cost driver in maintenance, OOP economically seems to me as a mistake when you have other techniques at your disposal.

When it was introduced more than 20 years ago the OO class brought encapsulation and polymorphism into the game which was beneficial. Thus, OOP was clearly better than procedural programming. But a lot of time has passed since then and we learned much about all the little and not-so-little flaws. It’s time to move on…

A better way

Now, that I have discredited the OO class as offered to you by Java, C# or other mainstream OOP languages I feel the duty to show you an alternative as it is available in Clojure and Haskell. It’s based on the following fundamentals:

Data is immutable, and any complex data structure (regardless of being more like a map or a list) behaves like a value: you can’t change it. But you can get new values that share a lot of data with what previously existed. The key to make this approach feasible are so-called “persistent data structures”. They combine immutability with efficiency.

Data is commonly structured using very few types of collections: in Clojure these are vectors, maps and sets, in Haskell you have lists, maps, sets and tuples. To combine a type with a map-like behaviour Clojure gives us the record. Haskell offers record syntax which is preferable over tuples to model more complex domain data. Resulting “instances” of records are immutable and their fields are public.

Functions are not part of records, although the Haskell compiler as well as the Clojure defrecord macro derive some related default implementations for formatting, parsing, ordering and comparing record values. In both languages we can nicely access pieces of these values and cheaply derive new values from existing ones. No one needs to implement constructors, getters, setters, toString, equals or hashCode.

Defining data structures

Here’s an example of some records defined in Haskell:

module Accounting where
data Address = Address { street :: String
                       , city :: String
                       , zipcode :: String} 
             deriving (Show, Read, Eq, Ord)

data Person = Person { name :: String
                     , address :: Address}
            deriving (Show, Read, Eq, Ord)

data Account = Account { owner :: Person
                       , entries :: [Int]}
             deriving (Show, Read, Eq, Ord)

And here’s what it looks like in (untyped) Clojure5:

(ns accounting)
(defrecord Address [street zipcode city])

(defrecord Person [name address])

(defrecord Account [owner entries])      

Let’s see what we now have got:

  • Thread-safe data.
  • Extensional equality.
  • Types.
  • No subtyping.
  • Reasonable default implementations of basic functionality.
  • Complex data based on very few composable collection types.

The last item can hardly be overrated: since we use only few common collection types we can immediatly apply a vast set of data transformation functions like map, filter, reduce/foldl and friends. It needs some practice, but it’ll take you to a whole new level of expressing data transformation logic.

But where should we implement individual domain logic?

Adding domain functionality

If we only want to implement functions that act on some type of data we can pass a value as (for example) first argument to those. To denote the relation between the record and the functions we implement these in the same Clojure namespace or Haskell module that the record definition lives in.

Let’s add a Haskell function that calculates the balance for an account (please note that the Account entries are a list of integers):

balance :: Account -> Int
balance (Account _ es) = foldl (+) 0 es

Now the Clojure equivalent:

(defn balance
  [{es :entries}]
  (reduce + es))

If we now want to call these functions in an OO like “subject verb” manner we would take Clojure’s thread-first macro ->. To get the balance of an account a we can use the expression (-> a balance). Or, given that p is a Person, the expression (-> p :address :city) allows us to retrieve the value of the city field.

In Haskell we need an additional function definition to introduce an operator that allows us to flip the order:

(-:) :: a -> (a -> b) -> b
x -: f = f x

Now we can calculate the balance for an account a using a-:balance, or access the city field with an expression p-:address-:city.

Please note, that the mechanisms allowing us to flip the function-argument-order are totally unrelated to records or any other kind of data structure.

Ok, that was easy. For implementing domain logic we now have

  • Ordinary functions taking the record as parameter.
  • A loose “belongs-to” relation between data and functions via namespace/module organization.
  • Independent “syntactic sugar” regarding how we can denote function application.
  • No hassle when we want to extend functionality related to the data without being able to alter the original namespace/module.

From an OO perspective this looks like static functions on data, something described by the term Anemic Domain Model. I can almost hear that inside any long-term OO practitioners head it starts to scream: “But this is wrong! You must combine data and functions, you must encapsulate this in an entity class!”

After doing this for more than 20 years, I just ask “Why? What’s the benefit? And how is this idea actually applied in thousands of systems? Does it serve us well?” My answer today is a clear “No, it doesn’t”. Actually keeping these things separate is simpler and makes the pieces more composable. Admittedly, to make it significantly better than procedural programming we have to add a few language features you have been missing in OO for so long. To see this read on, now the fun begins:

Polymorphism

Both languages allow bundling of function signatures to connect an abstraction with an API. Clojure calls these “protocols”, Haskell calls these “type classes”, both share noticable similarities with Java interfaces. But in contrast to Java, we can declare that types participate in these abstractions, regardless of whether the protocol/type class existed before the record/type declaration or not.

To illustrate this with our neat example, we introduce an abstraction that promises to give us somehow the total sum of some numeric data. Let’s start with the Clojure protocol:

(defprotocol Sum
  (totals [x]))

The corresponding type class in Haskell looks like that:

class Sum a where
  totals :: a -> Int

This lets us introduce polymorphic functions for types without touching the types, thus essentially solving the expression problem, because it let’s us apply existing functions that only rely on protocols/type classes to new types.

In Clojure we can extend the protocol to an existing type like so:

(extend-protocol Sum
  Account
  (totals [acc] (balance acc)))

And in Haskell we make an instance:

instance Sum Account where
  totals = balance

It should be easy to spot the similarity. In both cases we implement the missing functionality based on the concrete, existing type. We’re now able to apply totals on any Account instance because Account now supports the contract. What would you do for a similar effect in OO land?6

With this, the story of function implementation reuse becomes very different: protocols/type classes are a means to hide differences between concrete types. This allows us to create a large number of functions that rely solely on abstractions, not concrete types. Instead of each object carrying the methods of its class (and superclasses) around, we have independent functions over small abstractions. Each type of record can decide where it participates in, no matter what existed first. This drastically reduces the overall number of function implementations and thus the size of the resulting system.

Perhaps you now have a first idea of the power of the approach to polymorphism in Clojure and Haskell, but wait, polymorphism is still only a special case of dispatching function invocations.

Other ways for dispatching

Let’s first look from a conceptual perspective what dispatching is all about: you apply a function to some arguments and some mechanism decides which implementation is actually used. So the easiest way to build a runtime dispatcher in an imperative language is a switch statement. What’s the problem with this approach? It’s tedious to write and it conflates branching logic with the actual implementation of the case specific logic.

To come up with a better solution, Haskell and Clojure take very different approaches, but both excel what any OO programmer is commonly used to.

Haskell applies “Pattern Matching” to argument values, which essentially combines deconstruction of data structures, binding of symbols to values, and — here comes the dispatch — branching according to matched patterns in the data structure. It’s already powerful and it can be extended by using “guards” which represent additional conditions. I won’t go into the details here, but if you’re interested I recommend this chapter from “Learn You a Haskell for Great Good”.

Clojure “Multimethods” are completely different. Essentially they consist of two parts. One is a dispatch function, and the other part is a bunch of functions to be chosen from according to the dispatch value that the dispatch function returns for an invocation. The dispatch function can contain any calculation, and in addition it is possible to define hierarchies of dispatch values, similar to what can be done with type based multiple inheritance in OO languages like C++. Again very powerful, and if you’re interested in the details here’s an online excerpt of “Clojure in Action”.

By now, I hope you are able to see that Clojure and Haskell open the door to a different but more powerful way to design systems. The interesting features of the OO class like dispatching and ease of implementation reuse through inheritance fade in the light of more general concepts of todays FP languages.

To complete the picture here’s my discussion of some left-over minor features that OO classes provide, namely type parameters, encapsulation and intensional equality.

Type parameters

In case of Haskell, type parameters are available for types (used in so-called “type constructors”) and type classes. If you know Java Generics then the term “type variable” is a good match for “type parameter”. These can be combined with constraints to restrict the set of admissable types. So, with Haskell you don’t loose anything.

Clojure is basically untyped, therefore the whole concept doesn’t apply. This gives more freedom, but also more responsibility. However, Typed Clojure adds compile-time static type checking, if type annotations were provided. For functions, Typed Clojure offers type parameters. But, AFAIK, and by the time of this writing, Typed Clojure doesn’t offer type parameters for records.

Encapsulation

The OO practitioner will have noticed that the very important OO idea of visibility of members seems to be completely missing in Clojure and Haskell. Which is true regarding data.

Restriction of visibility in OO can have two reasons:

  • Foreign code must not rely on a members value because it’s not part of the API and it might be subject to change without warning.
  • Foreign code must not be allowed to change a members value when this could bring the object into an inconsistent state.

As available in todays mainstream OO languages, I don’t see that the first reason is sufficiently covered by technical visibility, because the notion of “foreign” can not be adequately encoded. In fact, an idea like PublishedInterface makes only sense because visibility does not express our intent. Instead of a technical restriction, sheer documentation seems to be a better way to express how reliable a piece of a data structure is.

Regarding the second reason, immutable data and referential transparency is certainly helpful. A system where most of the functions don’t rely on mutable state is less prone to failure caused by erroneous modification of some state. Of course, it is still possible to bring data into an inconsistent state and pass this to functions which signal an exception. But the same is possible for mutable objects, the only difference being that a setter signals an exception a bit earlier. Eventually validation must detect this at the earliest possible point in time to give feedback to the user or a external system. This is true regardless of the paradigm.

Regarding visibility of functions, Haskell as well as Clojure give us a means to make a subset of them available in other modules / namespaces and to protect all others from public usage. Due to its type system Haskell can additionally protect data structures from being used in foreign modules.

Intensional Equality

In a language that consistently uses value semantics there is only extensional equality and the test for equality works everywhere with = or ==. If we need identity in our data then we can resort to the same mechanism that we use for records in a relational database. Either define an id function that makes use of some of the fields or add a surrogate key. Anyway, in a system with minimal mutable state the appetite for object identity diminishes rapidly.

Are there any downsides?

Of course, apart from being different and requiring us to learn some stuff, programming without objects — especially without mutable state — sometimes calls for ideas and concepts that we would never see in OOP7. For example, there is the concept of a zipper to manipulate tree-like data which is dispensable in OO because we can just hold onto a node in a tree in order to modify it efficiently in place. Or, you can’t create cyclic dependencies in data structures. In my experience, you seldom encounter these restrictions, and either detect that your idea was crap anyway, or that there is an unexpected but elegant solution available.

A system without side-effects is a useless thing. So real-life functional languages offer ways to enable them. By being either forced by the compiler to wrap side-effects (as in Haskell) or by being punished by obstrusive API calls (as in Clojure) we put side-effects at specific places and handle them with great care. This affects the structure of the system. But it is a good thing, because having referential transparency in large parts of a code base is a breeze when it comes to testing, parallelization or simply understanding what code does.

You may argue that imperative programming leads to more efficient programs compared to functional programming. You’re right. Manual memory management is also likely to be more efficient than relying on a garbage collector. If done correctly. Is there anyone on the JVM missing manual memory management? No? So, there is certainly a price. But modern multi-core hardware as available in enterprise computing is not as expensive as precious programmer time.

Conclusion

This was really a long journey. I started by breaking down and analyzing the features that the good old OO class offers. I continued with showing how you can do the same and more using a modern FP language. I know, it looks strange and I promise it feels strange… but only in the beginning. In the end you’re much better off learning how to use the power of a modern language like Clojure or Haskell.


1. Here, I refer to OO concepts as available today in the mainstream. Initially, objects were planned to be more like autonomous cells that exchange messages, very similar to what the Actor model provides. So in a sense, what we think of today as OOP is not what its inventors thought it should be. 2. Which is of course only true, if OOP languages are all you know. Read about the Blub Paradox to get an idea why it is so hard to pick up totally different approaches to programming. 3. Regarding equality of data the term “ValueObject” is IMO an outright misnomer. A Value supports extensional equality whereas an Object has identity and therefore supports intensional equality. The term “Value” alone would have been better. 4. This “subject verb” order isn’t appropriate in every situation, as Steve Yegge enjoyably points out in his famous blog post Execution in the Kingdom of Nouns. 5. If you are concerned about missing type annotations in Clojure records: There exists an optional type system called Typed Clojure that allows to add compile-time static type checking. Alternatively you can use libraries like Domaintypes or Prismatic Schema to add type-like notation and get corresponding runtime validation. 6. What can we do in OO to add methods contained in a new interface to an existing type without touching the types implementation? Use the Adapter pattern, if you’re lucky to influence how instances are created, and if you’re able to retain a reasonable equality relation between wrapped and unwrapped instances. Good luck! 7. No, I don’t refer to Monads. Java 8 introduced Optional which is only a flavor of the Haskell Maybe Monad. So there is evidence that the idea is useful in OOP.

Written by youryblog

September 30, 2014 at 9:14 PM