Coding.Fitness

The Jet Engine we built in 2015.

Louie Bacaj — Thu, 04 Feb 2016 20:53:00 GMT

"Because it's probably harder to build a massive eCommerce platform than an actual Jet engine."

So you want to build a massive online eCommerce platform?

A lot of friends, and former colleagues, want to know what its been like to build Jet.com. I figured I would take a little time to talk about some of the things we've built as a team as well as reflect on the last year. I'd like to share some things that are working great, some of the things that aren't and what the game plan is for the new year; meaning what are we going to build next.

So although jet.com has had a colossal amount of media attention over the last year, we are still a pretty young company. Two years ago, this company didn't exist and less than seven month ago it hadn't launched and had under two hundred employees. Now we have over a thousand people and are growing at a blistering pace, with over two million shoppers on the platform. The media attention hasn't all been good and it's due in part to the staggering amount of money we have raised; it's made us as much a target as it has empowered us to deliver a tremendous amount of technological value.

That technological value is what I would like to talk about in this blog post and I will refer to it as the Jet Engine, which sounds a little cliché, but I assure you it's probably harder to build an innovative eCommerce platform in 2015-2016 that will scale to hundreds of millions of customers than it is to build an actual Jet Engine. (Full disclosure: I've never actually built an actual airplane engine, so who really knows which one is harder to build, I bet you the airplane engineers never built a massive eCommerce platform.)

The tech stack

The tech stack has been talked about quite a bit, in a few places. We are using Azure as our cloud provider and as the saying goes "when in Rome do as the Romans". We have decided to build on top of the .NET technology stack. This stack continues to and has undergone a radical transformation in the last couple of years. That transformation towards open source has been amazing. We are already starting to see some of those benefits as we prepare to run .NET Micro Services on Linux boxes with support from Microsoft; that alone is going to be something else. Like most fads in technology, Microsoft is starting to come back full circle again these days. With all of the open source commitments Microsoft has made lately, dare I say the stack might actually be pretty cool again.

Look... the CTO and the lead engineers that were here from day one, didn't decide to use .NET because it was cool or use it in the traditional ways. Meaning they didn't pick up this technology and use C# to build things the same old way, which is to say using object oriented principles. Those principles were popular during the era of single core single computer applications. No, they decided to build this engine for scale from the get go and for the era of distributed computing.

You might be asking what does that even mean? And rightfully so, it sounds outlandish even to me when I say it. But hear me out because I believe in this. Most startups build an MVP (minimum viable product) and as they grow, they realize they need more. Twitter, for example, is famous for building everything in Ruby, hitting a massive brick wall called scale and turning right around and writing things in Scala. Many of you reading this will know Scala is a functional language and that functional languages tend to be stateless and some are even immutable first. Twitter was able to overcome these scale problems not by throwing more hardware at them but by using the right tools, in this case for massive distributed systems Functional Languages are the right tool.

"...it sounds outlandish even to me when I say it but hear me out because I believe in this.... Twitter for example is famous for building everything in Ruby hitting a massive brick wall called scale....They were able to overcome these Scale problems ... by using the right tools...in this case... Functional Languages are the right tool."

Immutability at scale is really important because it allows you to go from 30k members to a staggering 2.5 Million+ and not even feel blip within 3 months, like we did here at Jet from the end of July to October. That's powerful! I'm not saying we couldn't do the same thing another way, but I am saying we would have needed more discipline to not fall back into the old traps of having state in our Micro Services. All of our Micro Services (or almost all) are stateless, which means you put the same thing in you get the same thing out, not once or twice but every single time. In the math world they call this referential integrity and we try to stick to it because it works well for scale.

I believe this was the most forward thinking decision made so far and the end of 2015, during the holiday season, proved to be absolutely true. I am not taking credit for this decision because when it was being made, I was a SOLID principles man at another company. But by my teammates made a great decision. Like most of the people reading this blog post, I didn't know about most of this before I joined jet.com. I learned it here at Jet and, more importantly, I now truly believe if you want to build for scale and do it fast you need to use right tools and F# proved to be the right tool for us. Whenever we build something we always consider scale as the top priority item.

"I believe [using F#] the most forward thinking decision made so far"

However, as it turns out, scale isn't the only good reason to use F#. The productivity it provides is unmatched and the lines of code needed to write something in F#, when compared to anything object oriented is something you can't compare. A fifty line Micro Service in F# may end up being hundreds of lines of code in any object oriented language, certainly true of Java or C# which I've used extensively professionally. So conciseness and the fantastic type system that pushes you toward correct easy to reason about code is another great reason why we picked F#. Besides the productivity, the massive amount of parallelism you get right out of the box with F# simply because you never need to worry about a mutex, a lock, or a race condition is something you can't even compare to in Object Oriented languages when writing fast code.

Azure as a cloud provider is also pretty damn good. Maybe at the moment not as good as AWS but that's not important. It's not that important because they are gaining ground every single day, and in some ways are even better for a price conscious startup. It is certainly comparable in feature for feature parity and at the end of the day every engineer knows it comes down to the code, not the hardware or the cloud provider (see my Twitter example above). As a company looking to grow to hundreds of millions of shoppers, we were looking for a cloud provider that is going to be around for a while and grow with us.

So I've spent a lot of time talking about F# as it is baked in our core. However, we leverage so many other technologies. Our front end uses Node.js as the entry API to the consumer side of things. Lots of our internal tools that are built in house for member services and retail partners use React, Angular 2, and many other front end technologies supported by F# services in the back. We even use C# and ASP.NET, dare I say maybe soon even some vNext stuff without IIS, in many cases supported by F# Micro Services that house most of the logic and are the work horses. We have some beautiful Android and iOS apps as well.

"[Jet] is an amazing place, trying to accomplish some pretty damn challenging and amazing things."

We use things like Redis for fast in memory cache, things like EventStore as immutable backend storage, HBase and HDFS, Azure SQL, Kafka and Azure Queues, Storm and even Spark. Among many other technologies that would be too much to talk about in this one blog post. The truth is each team will look into the things they are building and pick the best tools for the job. It's an amazing place, trying to accomplish some pretty damn challenging and amazing things.

Building this kind of engine is not all roses.

So some of the things that haven't worked so well on the tech stack side of things are mostly related to our pain points and our outages. We've had a few outages, the great thing is that none of them have really been related to scale. Almost all outages were short blips that got resolved fairly quickly. The one thing we are pretty proud of is that during the highest volume shopping season, the holidays, when other eCommerce shops were dropping like flies we had no outages.

Redis has caused us a few pain points. More specifically, if you could look at our internal post mortem's it's been Azure Redis. However, as we have evolved our usage of Redis we are starting to see much better uptime. EventStore is a great technology, an immutable data store that goes really nicely with an immutable functional language. You can never update an event or record, only add more to it, which is really nice given storage is cheap and holding onto things forever is a really nice thing to have. When we run EventStore on SSD's its fantastic but not great on the cloud where network attached disks tend to have lots of failures. In some ways we are also to blame for using it as a persistence layer and as a Bus at times when we were projecting off of it.

The real winner here for us in the technology side has been using F# in that it's infinitely scalable in terms of the level of parallelism and productivity. At the same time, it gives us the blistering speed we need to build the core of this engine, which is a realtime pricing algorithm that tries to find the consumer the most savings it can find by pulling all kinds of cost out of the supply chain in realtime.

The teams and the architecture.

One of the unique things about Jet, I think, is the amount of ownership they expect out of their people. That is to say engineers are expected to own the things they build, and business folks are expected to own the things they are working on. Teams build their Micro Services and architecture in a way they want and other teams consume from them or read from one of their immutable stream based systems. The level of ownership each team has serves as motivation and fosters an engineering first culture.

Ownership is very important to engineers, take a look at this post by Maxime where he explains what ownership means:

"No matter what’s your job, you don’t have a significant contribution on the game. You’re a drop in a glass of water, and as soon as you realize it, your ownership will evaporate in the sun. And without ownership, no motivation."

This engineering first culture we are striving for at Jet, I would argue is much like what Google has tried to create. This means that we as software engineers have to take pride in the things we build and are treated as owners in the business and we have to ensure the things we put out there in the world are of the highest quality.

As an example of this ownership; the pricing team which is known internally as Superman has a set of services and tools that they own. If you want to know about a price in realtime you ask Superman they are leveraging, the markets, the supply chain, and geographical factors among many other things to provide you a realtime price. You can get this by going to one of their RESTful end points, projecting from one of their realtime streams in Kafka. You have options like projecting that onto a database or an immutable data store. Obviously this is a powerful thing because if you want to get at this data programmatically you have multiple options. The ordering team, which is known internally as Gambit, similarly has a set of services and tools that they own. If you want to know about a members purchase history or look up some specific things about a user or order you have similar options.

The other important factor here is the folks building these features own those exact tools and services so that when things go wrong they are expected to fix them. There isn't some bug team for you to dump your code onto. You and your team own the code you're writing and it's speed, testing, and quality are reflection of you and your team’s abilities.

As another example there is a team, known internally as Flash, that is downstream consuming all of this massive amounts of data and trying to make sense of it. The data generated by all the other teams is aggregated and reporting is built on top of it in Flash. If you want reports or you want to answer tough business questions you go to Flash and if something is wrong then flash will fix it. Likewise the spiderman team provides data at scale and this data then informs the realtime pricing engine so that Jet is constantly trying to find you the cheapest price. There are dozens of teams doing extremely challenging and interesting things at Jet.com. I hope you are seeing the patterns here.

This type of technology and engine isn't trivial to build and each team has ownership and control of their architecture. An engineering driven company and culture is also not trivial to build. Areas where multiple teams are impacted are decided by a design review sessions that include all of the engineers and require all the teams to come to hash things out and come to an agreement.

There are many technology teams at Jet; they all own their destiny and respective architectures, tools, and technology decisions. If you land on the Nova team your worried about products titles, descriptions, the overall catalog and it's data quality. You will build heuristics and rules to clean the data. If you land on the batman team you're in charge of the front end and the consumer facing experience, and the first line of interaction with the user.

I am sure you've seen the pattern already in that each team is named after some Superhero. It probably sounds silly but we take this very seriously as our code bases are named in this way. If you look at Thor you will see one of the best Warehouse Management Systems that's been built internally at an ecommerce platform. The Professor X team is in charge of search and is dealing with optimizing that set of services and building interesting things on top of that.

Meanwhile if you happen to land on my team, Storm, you will be working on building the levers required for marketing to get things done. That means feeds, integrations, and lots of internal tools to give marketing the power they need to grow this Jet engine.

Jet.com at its core is a distributed, immutable, realtime system. At the very top level the architecture can be described as event sourced but it's much deeper than that. Building a near realtime distributed system has some challenges but coupled with our micro service architecture, F# as the core language, complexity has been greatly reduced. This means it's fairly easy and straight forward to add functionality. Which is a great thing given how fast this engine is moving and evolving.

What's the plan for going forward?

Given all of the exciting things we've already built, you'd be surprised to find out we have a ton of things still left to build. We will be focusing on better ingesting the massive amounts of data we are creating and focus on making that the lifeblood of our system.

Ultimately, we want to save the consumer as much money as possible by taking out as much supply chain cost as we can from the whole ecommerce experience. This isn't that easy to do. I think much of this has come together pretty nicely in the smart cart technology in that the more you shop the more prices drop. We will continue to grow this technology and iterate on it, make it more intuitive, make it more innovative and take into account all kinds of factors.

We will continue to leverage F# and try to build better personalization’s and recommendations. We will empower merchants and brands to flow promotions more seamlessly through our system rather than having to enter promotion codes and other old and frustrating mechanisms. Other things you can count on us doing is market to you like crazy in hopes of convincing you that Jet is the best place for you to shop.

Overall, Jet is simplicity on the side of a very complex industry. Jet started with the premise of pulling out as much cost as possible for the consumer and jet has the consumers back. The savings from not having brick and mortar stores isn't going to consumers in the current state of this industry, the savings that goes from shipping from the same warehouse and sending it all in one box isn't going to the consumer, unless of course you're shopping with Jet. So far we have demonstrated that this industry can have much more efficient systems. Our engine aims to save consumers lots of money. You can count on us improving and perfecting that part of the jet engine into 2016.

F# Powered Realtime Dashboard.

Louie Bacaj — Fri, 25 Dec 2015 15:06:00 GMT

Let's build a dashboard powered by .NET and JavaScript, specifically we will use F# and vanilla JavaScript with little bit of jQuery.

A while back I wrote an article that was quite popular on building a similar dashboard using Angular and C#, partly because people can't get enough dashboards and because I love this sort of thing, that article can be found on SitePoint.

This time the technology powering our dashboard will be similar, in that we are still using .NET and JavaScript and the web sockets library SignalR. The similarities will end there though, as this new dashboard will be significantly better. We will build it using a simple stateless service that is functional first. This service will power our dashboard on the server and use the Web Sockets library, SignalR, to move data in realtime to the client side.

If you would like to find out a bit more about SignalR, my colleague and friend Troy Kershaw, wrote up beautiful post that can be found here.

The source code for the dashboard we will be building can be found on GitHub.

The final product will look something like this

Some of the key benefits that will make this dashboard better in F# rather than writing it in C# and Angular are:

It will be extremely easy to reason about this code base.
It is almost completely stateless except for the bootstrapping needed to get Owin and SinglaR up and running.
Perhaps the most important reason of all to use F# over any OOP language Is the amount of code required to get the same exact thing up and running will be significantly less.
You should be able to get this up and running using completely open source and free software (Visual Studio Community edition, F#, and .NET core. You may even be able to run it on Linux, although my tutorial will be is easier to follow on Windows).

I will leave it up to you to tell me in the comments if it's better.

To be clear I am not trying to rail against one of my favorite languages, one that is near and dear to me C#, but I'm trying to prove a point that F# is really on another level of abstraction and works fantastically for general purpose programming, exactly like this sort of thing.

Let's get to it, shall we?

First thing is first, let's get an F# solution going for our server side code (Of course you can run all of this on your own Windows machine. I do know my good friend John Montaya likes to get these this sorts of things running on Linux using Mono but I haven’t tried that approach but you can, you will just have to change the code slightly to pull Linux metrics. As an IDE I will be using Visual Studio, simply because its easy enough and there is now a decent free version with Community Edition. Some of my former colleagues used to say its the Cadillac of IDEs despite what you may have heard. (I'm not really sure if anyone drives Cadillacs anymore honestly.)

Creating the projects

By Creating a Library we will have the option of referencing our code within some service or alternatively we can simply invoke it to start via it's FSX file and that’s the beauty of F#. We don’t even need a Console executable.

Let's add a few refrences for our SingalR to self host through Owin

Next up open up NuGet and add the following libraries either via command line or through the GUI.

Microsoft.AspNet.SignalR.Selfhost Microsoft.Owin.Cors Microsoft.AspNet.SignalR.Owin Fsharp.Dynamic

The last library will help with adding .NET dynamic features to F#. SignalR uses this and basically the libraries overload the ? operator.

Now that our server side project is ready to go let us create a very simple bare bones ASP.NET client app to consume our SignalR F# Server. You can do this via Node as well if you prefer, basically anything that can serve up some client side JavaScript and static files will work as a client.

I will be doing it right here in Visual Studio since its nice and easy to do so, if you wish to do the same right click on the main solution file -> click add New Project then select C# (Don't worry we wont be writing any C#!) we just need a bare bones web app with nothing in it.

Then select an empty ASP.NET project, basically something that can serve up our JavaScript and client code.

After creating the client project let's add a reference to the SignalR client using Nuget like we did before for the F# project, this will bring in the required jQuery and the rest of the JavaScript code to hook up to our F# SignalR Server. (note: you can just download the SignalR references from their site if you dont want to use asp.net as the client.)

Microsoft.AspNet.SignalR.JS

Add an index.html file as well to the project by right click -> Add -> New Item -> select HTML page

I went ahead and renamed those default files in our F# project and it should all look something like this.

We have our project ready to go, lets write some damn code!

Next we will wire up our SignalR.

We first need to create a simple Startup.fs file that will bootstrap SignalR via Owin and start it up as a self hosted app without IIS. Normally code like this would be auto generated in most C# SignalR projects but F# we have to do this ourselves, a minor price to pay for the rest of the benefits we get.

module Startup

open Microsoft.AspNet.SignalR  
open Microsoft.AspNet.SignalR.Hubs  
open Microsoft.Owin.Hosting  
open Microsoft.AspNet.SignalR.Owin

type public Startup() =  
    member public this.Configuration(app:Owin.IAppBuilder) = 
        let config = new HubConfiguration(EnableDetailedErrors = true)
        Owin.MapExtensions.Map(app, "/signalr",
            fun map -> 
                Owin.CorsExtensions.UseCors(map, Microsoft.Owin.Cors.CorsOptions.AllowAll) |> ignore
                Owin.OwinExtensions.RunSignalR(map, config)) |> ignore

        ()

SignalR uses hubs so the next thing we need to create is a hub that we can use to call clients and also so that we can make calls and keep track of who is connecting, should we need to.

module MetricsHub

open System  
open Microsoft.AspNet.SignalR  
open Microsoft.AspNet.SignalR.Hubs  
open Microsoft.Owin.Hosting  
open Microsoft.AspNet.SignalR.Owin  
open FSharp.Interop.Dynamic


    []
    type MetricsHub() = 
        inherit Hub()

        //if we were intrested in seeing who is connecting
        //or doing something on a new connection this would be the place
        override x.OnConnected() =
            base.OnConnected()

        // A function that can be invoked by any client since signalr uses web sockets for two way communication.
        member public x.SendMessage(message : string) : unit =
            base.Clients.All?addMessage (message)

The main entry of the application, our DashStart.fsx can look like the following to spin up our SignalR server.

#I "bin"

#r "Debug\Newtonsoft.Json.dll"
#r "Debug\Microsoft.AspNet.SignalR.Core.dll"
#r "Debug\Microsoft.AspNet.SignalR.Owin.dll"
#r "Debug\Microsoft.Owin.Cors.dll"
#r "Debug\Microsoft.Owin.dll"
#r "Debug\Microsoft.Owin.Hosting.dll"
#r "Debug\Owin.dll"
#r "Debug\FSharp.Interop.Dynamic.dll"
#r "Debug\Dynamitey.dll"

#r "Debug\LouiesGuiAdventDash.dll"

open Owin  
open System  
open FSharp.Interop.Dynamic  
open Microsoft.AspNet.SignalR  
open Microsoft.AspNet.SignalR.Owin  
open Microsoft.Owin.Hosting  
open Microsoft.Owin.Cors  
open Microsoft.AspNet.SignalR.Hubs  
open Startup  
open MetricsHub  
open LouiesGuiAdventDash


let url = "http://localhost:8085/"

AdventDash.start url  
0

Now we need to flesh out our AdventDash.fs file which contains the meat and potatoes of our server. We can start with something like this:

namespace LouiesGuiAdventDash

module AdventDash =  
    open System
    open System.Threading
    open Owin
    open FSharp.Interop.Dynamic
    open Microsoft.AspNet.SignalR
    open Microsoft.Owin.Hosting
    open Microsoft.Owin.Cors
    open Microsoft.AspNet.SignalR.Hubs
    open Microsoft.AspNet.SignalR.Owin
    open Startup
    open MetricsHub

    //keep it running
    let private signal = new ManualResetEventSlim(false)
    let wait() = signal.Wait()

    let start (signalrEndpoint:string) =

        try            
            use webApp = WebApp.Start(signalrEndpoint)
            let context : IHubContext = GlobalHost.ConnectionManager.GetHubContext()
            for x in [ 0..1000 ] do
                System.Threading.Thread.Sleep(1000)
                printfn "Server Sending Value to Client %s: " (x.ToString())
                context.Clients.All?addMessage (x.ToString())

            printfn "running..."
            printfn "listening on %s"  signalrEndpoint
            wait()
        with 
        | ex -> 
            printfn "%A" ex
        ()

If we try running this via the FSX file either through visual studio or command line we will see the app start up and broadcast some dumb numbers, so we know at least we have wired everything correctly.

To push some real metrics about our server let's start pulling some Windows Specific performance counters, if you were on Linux you would have to pull those via Mono so this code would change. First let us model what that data will look like as an F# type, the serialization will automatically be handled for us. Let's add a performance model file PerfModel.fs

module PerfModel  
    open Newtonsoft.Json

    type PerfModel = {
        []
        MachineName   : string
        []
        CategoryName  : string 
        []
        CounterName   : string 
        []
        InstanceName  : string 
        []
        Value         : double 
    }

Now let's re-write our AdventDash.fs to have some REAL meat and potatoes, after all it is Christmas.

namespace LouiesGuiAdventDash

module AdventDash =  
    open System
    open System.Threading
    open Owin
    open FSharp.Interop.Dynamic
    open Microsoft.AspNet.SignalR
    open Microsoft.Owin.Hosting
    open Microsoft.Owin.Cors
    open Microsoft.AspNet.SignalR.Hubs
    open Microsoft.AspNet.SignalR.Owin
    open Startup
    open MetricsHub
    open System.Diagnostics
    open PerfModel

    //the context
    let interval = 1000
    let context : IHubContext = GlobalHost.ConnectionManager.GetHubContext()

    ///get the processs instance name via provided process id
    let getProcessInstanceName (pid: int) =
            let cat = new PerformanceCounterCategory("Process");
            let instances = cat.GetInstanceNames()
            instances 
            |> Array.filter(fun i -> 
                use cnt = new PerformanceCounter("Process","ID Process", i, true)
                let intval = (int)cnt.RawValue;
                if pid = intval then
                    true
                else false)
            |> Seq.head

    ///get the current running process name
    let getCurrentProcessInstanceName =
            let proc = Process.GetCurrentProcess();
            let pid = proc.Id;
            getProcessInstanceName pid

    ///set a few service counters so we can track some basic metrics
    let serviceCounters = 
        [
          new PerformanceCounter("Processor Information", "% Processor Time", "_Total")
          new PerformanceCounter("Memory", "Available MBytes")
          new PerformanceCounter("Process", "% Processor Time", getCurrentProcessInstanceName, true)
          new PerformanceCounter("Process", "Working Set", getCurrentProcessInstanceName, true)
        ]

    ///grab performance metric values and map them to our over the wire F# perfModel
    let metricsWorkflow (context : IHubContext) = async {
        let mappedCounters = serviceCounters
                                |> Seq.map(fun x ->
                                    {
                                      PerfModel.PerfModel.MachineName = x.MachineName
                                      CategoryName = x.CategoryName
                                      CounterName = x.CounterName
                                      InstanceName = x.InstanceName
                                      Value = (double)x.RawValue

                                    }
                                )
        //broadcast all of the metrics
        mappedCounters
        |> Seq.iter(fun perfModel ->
            printfn "CategoryName: %s CounterName:%s Value:%A" perfModel.CategoryName perfModel.CounterName perfModel.Value
        )
        context.Clients.All?broadcastPerformance(mappedCounters);
    }

    ///lets recurse infitly to broadcast our metrics 
    //we can do a while loop or any other mechanism we like
    let rec iBroadcast() = async {
        do! metricsWorkflow context
        do! Async.Sleep 1000
        return! iBroadcast()
    }

    let start (signalrEndpoint:string)  =
        printfn "starting..."
        try            
            use webApp = WebApp.Start(signalrEndpoint)
            printfn "running..."
            printfn "listening on %s"  signalrEndpoint

            iBroadcast() |> Async.RunSynchronously
        with 
        | ex -> 
            printfn "%A" ex
        ()

We can now run the server side code and see metrics being fired at a really nice steady rate.

A quick side note.

We can skip over this section this is just a few things about F# in general that I think are useful to know, for people who are new to this.

So this is not intended to be a plug about F#, but F# has had Async computations long before C# got them. That set of code that now exists in C#, you know the Async-Await was in F# first and it's always been simple to reason about and is still one of the best things about F#.

let! fSharpIsAwesome = async {  
 //do asynchronous computations
 //offload IO/network bound work here
 //don't forget everything is immutable and that's what makes this so beautiful!
}

Although this dash dosent make much use of real asynchronous computations its really easy to imagine some of this data coming from a database, or some other backend store. At that point and time these little async units of work would be a life saver.

One of our primary reasons to go with F# is the massive parallelism and concurrency we can get out of the language without having to worry about locks or those nasty race conditions that used to keep us up at night when we used to write Java and C#.

Look it isn't a coincidence that every company doing things at internet scale is moving towards functional languages. As much as I love my OOP and SOLID principles, those languages and design patterns were built in a world with single CPU single thread in mind. Today, in the age of Internet and cloud, we need something more scalable, we need something better.

At my current employer, jet.com, F# is the answer to the scale problem, it's why we moved from 30k members to 2 million within 4 months without ever feeling it. It was because of the forward thinking technology stack, I firmly believe that anything with state would have fallen over already.

The great thing about F# is that things are immutable first so we don't have to worry about race conditions or data getting changed on us by these threads. We will only worry about this sort of thing if we explicitly make things mutable.

Back to the tutorial - let's fix up our client code.

First thing is first, let's add some CSS to this thing. My brother, Anton Bacaj, who is on this blog and is a great full stack developer wrote up a good CSS layout for a dashboard.

Figure now is a good time to open source this thing - I will let him post up the SASS files and such but for now just grab the main.css and add it to the project it can be found here.

Let us add epoch.js which is a realtime charting library in JavaScript which can be found here. This relies on D3.js which we can just grab via Nuget. The idea here is just to show the data but we can wire this up to any charting library.

CORS is also an issue that pops up with these sorts of things so lets allow it on our client app as well via the following in our web.config

Next up let us create a new file in our scripts folder called signalrapp.js, this file will house our signalr connection and we can map our JSON data to our charts.

Here is what that file may look like for us, this is a very simple wiring without much logic except to format the data. The idea here is that we could use any charting library, and in fact there are much better ones these days

$(function () {

    $.connection.hub.url = "http://localhost:8085/signalr";
    var perfHub = $.connection.metricsHub;

    //setup some charts

    var lineChart = $('#lineChart').epoch({
        type: 'time.line',
        data: generateLineData(),
        axes: ['left', 'bottom', 'right']
    });

    var barChart = $('#barChart').epoch({
        type: 'time.bar',
        data: generateLineData(),
        axes: ['bottom', 'right']
    });

    var areaChart = $('#areaChart').epoch({
        type: 'time.area',
        data: generateLineData(),
        axes: ['left', 'bottom', 'right']
    });

    var charts = [
        $('#gaugeChart').epoch({ type: 'time.gauge', value: 0.5 }),
        $('#gaugeChart2').epoch({ type: 'time.gauge', value: 0.5 }),
        $('#gaugeChart3').epoch({ type: 'time.gauge', value: 0.5 })
    ];

    perfHub.client.broadcastPerformance = function (data) {
        var timestamp = ((new Date()).getTime() / 1000) | 0;

        var entry = [];

        data.forEach(function (dataItem) {

            if(dataItem.categoryName == "Processor Information")
            {
                console.log('processor: ');
                console.log(dataItem);
                entry.push({ time: timestamp, y: (dataItem.value / 1000000000000) });
                charts[2].update((dataItem.value / 10000000000000))

            }
            else if (dataItem.categoryName == "Memory")
            {
                console.log(dataItem);
                charts[0].update((dataItem.value / 10000))

                entry.push({ time: timestamp, y: (dataItem.value / 10000) });

            }
            else if(dataItem.categoryName == "Process")
            {

                if(dataItem.counterName == "% Processor Time")
                {
                    console.log(dataItem);
                }
                if(dataItem.counterName == "Working Set")
                {
                    console.log(dataItem);
                    charts[1].update((dataItem.value / 1000000000))
                    entry.push({ time: timestamp, y: (dataItem.value / 1000000000) });
                }
            }
        })
        lineChart.push(entry);
        barChart.push(entry);
        areaChart.push(entry);
    };

    $.connection.hub.start().done();

    //util function to generate some random data
    function generateLineData() {
        var data1 = [{ label: 'Layer 1', values: [] }];
        for (var i = 0; i <= 128; i++) {
            var x = 20 * (i / 128) - 10,
                y = Math.cos(x) * x;
            data1[0].values.push({ x: x, y: y });
        }
        var data2 = [
            { label: 'Layer 1', values: [] },
            { label: 'Layer 2', values: [] },
            { label: 'Layer 3', values: [] }
        ];
        for (var i = 0; i < 256; i++) {
            var x = 40 * (i / 256) - 20;
            data2[0].values.push({ x: x, y: Math.sin(x) * (x / 4) });
            data2[1].values.push({ x: x, y: Math.cos(x) * (x / Math.PI) });
            data2[2].values.push({ x: x, y: Math.sin(x) * (x / 2) });
        }
        return data2;
    }
});

Finally our index.htmlfile will look something like this. The metrics may not make much sense and we can think of much more interesting metrics to push and chart out but this is just to demonstrate the power of F# to build out these sorts of things.

  
  
  
    
    
    Louie's Advent Dash
    
    
    
    
    
    
    
    
    
    
    
  
  
    
        
            
                
                    
                        
                            
                        
                    


                    
                        Advent Dash
                    

                    
                        
                            
                        
                    
                
            
        
        



            

                
                    
                        
                            
                                
                                    
                                
                                
                                    Overall CPU Meters
                                
                                
                                    
                                
                            
                            
                                
                            
                        
                    

                    
                        
                            
                                
                                    
                                
                                
                                    Overall Memory Usage
                                
                                
                                    
                                
                            
                            
                                
                            
                        
                    

                    
                        
                            
                                
                                    
                                
                                
                                    App Memory Usage
                                
                                
                                    
                                
                            
                            
                                
                            
                        
                    
                    
                        
                            
                                
                                    
                                
                                
                                    CPU / Memory
                                
                                
                                    
                                
                            
                            
                                
                            
                        
                    
                    
                        
                            
                                
                                    
                                
                                
                                    CPU / Memory
                                
                                
                                    
                                
                            
                            

                                
                            
                        
                    

                    
                        
                            
                                
                                    
                                
                                
                                    CPU / Memory

Let's spin it all up and see what it looks like:

We can substitute another charting library and the way I am consuming the metrics on the JavaScript side isn't the cleanest but really this post is about F# and the Advent Calendar not JavaScript!

Much can be said about the power of F# as a functional first language. There really are so many great benefits to using F#, that is why Jet.com uses it as our primary backend language to our distributed system and all of our micro-services and libraries are built in it. However, in this article I hope I have been able to show you the power of F# as a general purpose language, potentially, and hopefully, replacing even your favorite OO language for your day to day work.

Merry Christmas!

P.S. Don't forget to subscribe to the e-mail list below.

Startups as just a "buzz" word for engineers.

Louie Bacaj — Thu, 08 Oct 2015 22:03:00 GMT

"Startups right now are nothing more than a 'fad' a 'buzz' word in the tech industry. Get yourself a job at a well established company so you can learn something and have a job tomorrow."

A friend of mine and former colleague made that comment to me recently. He isn't unique in his thinking see below on a comment I saw on this startup advertisement.

As surprising as it may sound, many people with years of experience seem to have this notion that startups aren't worth the Risk/Reward. Leaving only the "kids" to go to startups.

I would like to tell you that startups are absolutely worth the Risk precisely because the rewards are not only quantifiable by money. The notion that they are just a 'fad' is so far removed from the truth and reality, you have no idea. My argument and advice is this; Startups are the best thing that could have happened to software engineers and if you could get into one you should immediately.

Now here is why I think that startups are great.

I truly believe that working at a startup will excel any engineers career trajectory by years. Three months after joining jet.com, a startup in e-commerce, I was invited to give a talk to the Iona College Computer Science Undergraduate student body and in that talk I mentioned that

"I have learned more in three months at Jet.com than I had in over a year at the Bank".

This isn't some gimmicky sales pitch, I'm not selling anything, I'm simply telling you that I have learned a tremendous amount working at a startup and you probably will too. Regardless how far along you are in your career.

In the three months that I had given that talk I had 6 micro services in production, six months later I have over 20 with four other apps that support the business to boot, running in a cloud environment with alerts, logging, dashboards and supported by better than enterprise grade libraries written internally at Jet.com.

I have had the opportunity to learn and use technologies such as Azure and most of its cloud offering. Tech like Redis, Elastic Search, immutable data stores such as Event Store, built lots of RESTful APIs written countless lines of code in a functional language called F#, in OOP languages like C# and used lots of JavaScript frameworks and even Node. Build tons of Micro Services. Worked in a stream based system with hundreds of millions of real time events and managed some fairly large near realtime feeds for partners like Google, Microsoft, and Facebook. Learned about digital Marketing, Retargeting, the different digital and traditional channels that exist in that space. Learned about stream and batch processing technologies such as Storm and Kafka and among many other things that would take very long to detail in this short opinionated blog post.

The point being get yourself into a startup as soon as you can.

As an example of the learning.

We have a feed with millions of price changes that goes out to our partners and the CEO of our jet.com wanted to A/B pricing strategies and see what performs better in our advertisments. At the bank that would have been a six month project, to modify a working feed that pushes out such high volume data. At a startup that was a two day thing.

Essentially the task was to Take this massive near realtime feed and build a cache with a front end so business folks can easily add products for A/B tests. At a startup the solution is yours to build, the problems are challenging but trust yourself you will solve them.

To finish off my example, I took this feed as it passes to our partners then I throw a few branch statements and look in this cache for marked products and finally alter their price and landing pages. This isn't your grand daddy's branch statement though, it's a branch statements on hundreds of million of product updates throughout the day. In F# we use pattern matching to branch and there are several advantages to this. Once I have the A/B test products I can then get the new prices and links to our advertising partners to see how they preform.

This was tested in QA within a day and deployed to production that same day. That test ran for the entirety of the next day. It was extremely successful and confirmed the hunch that displaying prices a certain way would in fact have a significant impact in raising the gross merchandising value (GMV) of the whole company and increasing conversion rates by seventy to ninety percent.

I would have never been able to build something that fast at a bank or any other company I worked at before. Not because adding a few branch statements is hard, but because no one would have ever moved so quickly, only a startup. No one would have trusted an engineer and one Marketing Lead, my good friend and colleague David Wu, to just build that within a few days and get it out into the world.

Startups move fast and that means you learn fast, trust me you have no choice.

The technology landscape has changed more in the last few years than it did in the whole thirty years before that, and it will continue to evolve. The big established corporations like the Banks are not the ones innovating, I promise no matter how good you are or how senior you are you will never get to deploy the kind of code you would have deployed at startup into production and have that level of business impact at a Bank. As much as I loved my Job at SunGard, because of the people I worked with and the responsibility they trusted me with, I would have never been able to make that kind of dent in a company with 17,000 people.

So if corporations like the banks and large enterprise level software companies like SunGard are not innovating who is? Is Google and Facebook? Will you learn more there than at a startup? Not really, No, at least I don't think so. In my opinion Google and Facebook, as innovative as they might have been at one point and as much open source software as they have committed, the reality is they have overcome their challenges. Their software is built, they need you to be a cog in their large machine now. Not a thinking problem solver that can toy around and experiment with this or that to see if it will work for them.

See this excellent article on the verge about Apple, Google, and Microsoft now solving the same problems

"Many of the brilliant minds working at Apple, Google, and Microsoft have found themselves having to fly the coop in order to give full expression to their creativity and embrace the inherent risk of that task. "

At startups they are trying to solve brand new problems. They will let you build tools to solve their problems all the time, internal tools, external tools, tools and libraries that get open sourced, eventually.

Startups are the only place you will be able to experiment, learn and keep up with the blistering pace of technology changes. Technology is going through some fundamental shifts, it always does at certain points in time, this time though is different. It's not really that different but the recent shifts to cloud big data and every other buzz word out there has had some effects and because you work at a large corporation you're missing out on all of it.

Some of the argument against startups.

I've worked at some established corporations in my past so I think I know what the argument against startups is. As in sure you've guessed two of my previous employers were SunGard and Bank of America, these were fairly large enterprises. That alone doesn't make me an expert in the matter, by any means, but it does lend some value and credibility to my argument, given that I now work at a startup.

The single biggest argument against working at a startup, that I have heard, is that there is inherent risk of it going under. To be fair many do. The startup might go under for not getting funding among many other reasons. I am arguing that None of those reasons really matters. You would have learned so much and will become so valuable that it doesn't matter, the risk is well worth it. From my experience the safety net you get from a large corporation is no safety net at all. Even large corporations let people go all the time, even engineers.

The learning you will do at a startup overshadows any risk and will ultimately makes you smarter, will propel your career forward because of the opportunities that are there if things go well, and will ultimately make you more valuable to any organization large or small because of all the learning you will get out of it even if things don't go well.

Finding a startup.

Look for many of my friends and former colleagues it is not that easy going from being an enterprise software engineer to working at a startup. I had challenges; such as most startups didn't even want me at first. I used too much .NET and Java, not enough Ruby or Python. Maybe I wasn't able to, or willing to solve their puzzle and riddles. Once you get over that And if you can get yourself into one you absolutely should. It's worth all the risk, the pay cut you might have to take, the learning is so worth it.

You won't be browsing Facebook at a startup and you won't get to keep up with what your friends are tweeting about. That's because you'll be working you're ass off and learning like crazy. If you want to make yourself invaluable get ahead on all real tech "fads" that are going to become standard in a few years at all the large corporations out there, then get yourself a job at a startup.

F# for the backend.

Louie Bacaj — Thu, 23 Jul 2015 23:55:00 GMT

F# your backend.

Functional languages are the future of our industry. If you're an object oriented developer looking for a modern Functional language to transition to and further your engineering skills, then I think you should strongly consider learning F#. I predict that within a few years almost all back end development, and possibly even front end, will happen in a functional language, or in functional style; not in an object oriented imperative way that we have grown so accustomed to.

It's no coincidence that we have seen so many extremly high tech companies, that make software for millions of people, transition to languages such as Scala, Haskell, more recently F#, and other functional languages. They all start out with an OO language like Ruby, Java, or Python, and eventually end up switching to something more "scalable". They have big problems and the only way to solve them is to massively distribute and parallelize.

It won't be long before that same computing environment exists, not just for big companies, but on your own devices, you know lots of cores and memory to take advantage of. But wait... that is already your computing reality, even your cell phone has 4 cores. As Brian Holley explains, in this excellent post, you must be this tall to write multi-threaded code. "If the fastest chips have N cores, a mostly-single-threaded program can only harness (1/N)th of the available resources. As N grows (and it is growing), parallel programs will win. We want to win, but we’ll never get there if it’s rocket science". Concurrency in the imperative languages we use right now is like rocket science. "Buggy multi-threaded code creates race conditions, which are the most dangerous and time-consuming class of bugs in software."

It's not just companies, though, even the languages themselves are moving toward this reality, with Java 8 adopting Lambdas, C# and .NET had LINQ and other functional constructs for some time as well. We even see JavaScript libraries leaning more and more toward the functional style of the language. The rise in popularity of functional first languages, like Scala, Haskell, and now F#, are signs that the languages we use and the industry as a whole are moving towards this paradigm.

Why my outlook has changed.

I've recently found myself doing a lot of functional development in F# and I thought I would share a few thoughts.

First of all, I must confess that I have always been a SOLID guy, within reason of course. Although they taught us what functional languages are about in school, it always seemed like those languages were reserved for the mathematicians among us and not the computer scientists. Perhaps I even saw some of the benefits of functional constructs and abstractions with LINQ while using C# and .NET over the last few years. Still, I would have never made a radical move like going all functional for all the things I'm working on. That is until I joined Jet.com, whom uses F# as the primary language, and it gets used in some very advanced ways.

Why Functional?

Functional first languages seem to do several things better. The first thing they do better is control state and limit unrestricted side effects. State and mutability are things we rarely think about in imperative object oriented languages, but they are the main reason for our headaches in today's computing world. Perhaps on a single machine with a single thread running a single app it isn't that big of deal to reason about OO code. Most of our OOP languages were made popular in that era, while functional languages were shunned. In an era where the computing reality was just that, things have changed, especially recently.

Perhaps on a single machine with a single thread running a single app it isn't that big of deal to reason about OO code.

Reasoning about state on a single app, single thread, single everything isn't that difficult and dealing with objects feels so natural. The challenges come in when you massively parallelize some work across a bunch of distributed compute nodes on the cloud. In imperative OO languages we need to jump through hoops because, after all, side effects exist by design and we need to ensure order with constructs like locks, semaphores or a mutex. The worst part is when something goes wrong; when some random bug, that only happens on the weekend when we aren't around, shows up in our code.

In those situations we need to try to recreate that state to reproduce the bug, which is very hard to do but it is most certainly due to some race condition or God knows what made its way into our code. Not to mention that this reliance on state is one of the largest contributors to system complexity. It is not that we're forbidden to use state, rather we want to contain it and use mutability in a very controlled way.

Don't get me wrong, Test Driven Development has made great strides towards eliminating bugs in our code. We are kidding ourselves, though, if we believe that TDD, BDD, or whatever tests we can concoct will eliminate those kinds of bugs completely. They still pop up and are very difficult to deal with especially when scaling in OO languages.

Getting started with F#

So it isn't the easiest thing in the world to go from an OO mindset to a functional one. F# helps by easing you into it. You can still make objects but you should really use the record types which are immutable, meaning you can't change them after you've made them. You could also make other things mutable by explicitly adding some syntax to let that happen, but you should really never do that, at least not when you first start learning the language because it eliminates all the benefits provided to you by this style of programming. So if you made a glass that is half full in your functional code, it is half full regardless which thread wants to see that glass and will never be half empty and you don't need to put a lock on the glass to make sure that's the case :)

The biggest challenge, at least for me, was going from C# to F# or from an imperative style to a functional one is not learning the syntax, because that part is easy, but rather it is changing your mindset and relying on functional constructs rather than object oriented ones. You will want to do things the same old way using loops and changing state but you should avoid that like the plague, at least the changing state part, so you can take full advantage of the functional style of programming. I must admit that by avoiding the things that come so naturally in imperative languages, I felt handicapped like someone was holding me back or tying my strong arm behind my back in the middle of a fight. I fought through that feeling and I encourage you to do the same and when using a functional language use it correctly, as they say "when in Rome do as the Romans".

In a very short period you will see the benefits of abstractions, composition, conciseness, correctness and other key benefits functional languages provide. You will have this feeling that you understand what your code is doing regardless of how many threads it's running on and you will get that not from running smoke tests, integration tests, or unit tests, but just by looking at it. I think once you get a taste of that, you will appreciate the value functional languages like F# provide and you will want to learn more.

Material to get started.

I read books, articles, and blog posts that help computer scientists leverage functional languages. I avoided material made by mathematicians talking about things like functors, monads, or monoids. If you are an imperative programmer used to an OO style, I think you should do the same, it will take a lot of the complexity away. It turns out those are simple concepts with crazy names because mathematicians got to name them. In practice, they are simple concepts and although different from what I was used to once I got over that initial hurdle it was smooth sailing.

Here are a few of those things I mentioned that worked for me to learn F#.

F# For Fun and profit - this is by far the best resource. It is in fact a book that is disguised as a website made easily digestible and God bless Scott for writing this because it is by far one of the best sources to learn F#.
There is a great list over at http://fsharp.org/. One of my favorite is the book Real World functional programming.
Work at a company that uses F# ;-)

I have found F# to be very friendly to the imperative programmer trying to turn a new leaf. The simplicity, conciseness, correctness, and power of abstraction you learn after just a short period is amazing. In three months time I feel very productive and capable in the language and while I miss the familiarity of my favorite OO languages like C# and Java, I don't miss the crazy race conditions and bugs that used to keep me up at night.

P.S. If you are at all intrested in joining jet.com, use my employee promo code LBACAJ and get a 12 month membership free, that usually costs 50 dollars, instead of the standard 3 months.

Let's "Marry Up" Angular to .NET

Louie Bacaj — Sat, 18 Oct 2014 21:50:00 GMT

UPDATE: That article on SitePoint was finally published, go there to see the two in action!

AngularJS and .NET seem to work great when used together. At first this type of technology arrangement may seem like overkill to .NET developers or perhaps .NET is not "cool" enough to Angular Developers. However, I would like to convince you that in fact pairing up AngularJS to .NET is a very good idea.

Back at my old employer, SunGard, we often spoke about "marrying up" a business analyst and a software engineer. The two compliment each other so well and rarely get in each other's way. The understanding of finance, brought in by a business analyst, would ensure the code that was developed by the engineer hit the mark. This idea isn't new, but it's rare to find one business analyst per engineer on a project. Most companies and people belive this to be counterintuitive and wasteful. This model, in fact, was so successful that our group at SunGard had an extremely great track record of delivering in an industry that is rife with failed projects. I was very fortunate to have been a part of that team. I think in alot of ways joining up Angular and .NET seems counterintuitive, I know it did to me in the beginning.

More recently, however, I have been utterly amazed how two, seemingly very different things, can compliment each other so well. I started building some side projects in Angular in an attempt to eat my own dog food; "All software engineers should learn JavaScript". One of these projects I am working on will be open sourced and available soon as a tutorial on SitePoint.

Live Demo of This in Action!

Here is a sneak-peek at the app I am building as a tutorial for SitePoint.com which should be published soon with all the source code. (Yes this data is really coming from the server through the SignalR and being consumed by Angular, still an early build though.)

The Client - AngularJs Try the Client Live Here.

The Server - .NET/Web Api 2.0/SignalR Try the Server Live Here.

In the same way that a business analyst would compliment an engineer I think AngularJS compliments .NET and C#. There are several reasons for this, some of which are extremely obvious, but I will mention anyway, and some are not so obvious that you only find when working with the two.

Why should Angular marry .NET?

With the advent of the Micro Service and the idea that code is debt, it is obvious that the huge monolithic code bases of yesterday have to go. C# and .NET have come a long way forward to help support this Micro Service architecture through Web Api 2.0+ and SignalR.

In the Windows server world, you will be hard pressed to find a better technology than .NET for development. This is because the framework is highly evolved and has constantly adapted to the latest trends and is proven enterprise grade. I am very fond of Java and the JVM languages because of their low cost to get started, but they are far behind and adding features now that .NET has had for years.

With Web Api and the SignalR libraries .NET 4.5 has really pushed the envelope for both easy to build and easy to maintain server side RESTful APIs and non-blocking asynchronous real time services.

While Google Go and NodeJS may very much be pioneers of the small code base that you don't have to feel bad about throwing away later on. Many people dont realize that C# and .Net have made great strides towards this Micro Service model especially with .NET 4.5. Both Go and Node are great, but they lack the library maturity, tooling, and deep OS integration that .NET commands on the Windows side.

In short, if you are going to build cross platform modern client side applications using AngularJS and host your Micro Services on a Windows server in house or on a Azure cloud instance, then it is going to be hard to find a better technology than .NET to pair up with AngularJS.

All well and good but why should .NET marry Angular?

It is true that .NET really shines brightly in the Windows server space. It is also just as true that you can have client side .NET applications fairly easily through WPF or Winforms. However, those technologies are reliant on the framework being installed and present on the client, which most mobile and web first devices just don't have support for. Angular on the other hand relies on HTML 5 and JavaScript being present, which basically means it runs on anything with a browser.

You might be saying, we have ASP.NET MVC 5! Well we do but that is primarily a server side first technology. ASP.NET is great for web applications it is lacking for client side applications. Since the server does all the lifting there is no single page application support. ASP.NET has no way of being wrapped into a good looking mobile app and because of that it is inadequate for the type of applications that today's world demands.

AngularJS, on the other hand, is completly reliant on client machines. This means that it can be easily wrapped using PhoneGap, Cordova, and Node WebKit. Then subsequently deployed to many mobile operating systems, like Android and iOS. AngularJS is giving us what .NET and Java promised years ago but never delivered; true cross platform support.

AngularJS can easily be developed in Visual Studio and can provide all kinds of bindings to .NET Micro Services. AngularJS provides flexibility of use with ASP.NET, one can chose to do the routing in Angular or in ASP.NET controllers. So you can use as much or as little as you want of either.

The feature list for Angular is amazingly large. Two way data binding which works like magic, directives, etc. For a more concise perspective on why AngularJS is great head over to SitePoint.com and read "Why I Love AngularJS and You Should Too".

The other not so obvious benefits of this relationship.

Angular forces great application development practices right out of the box. Everything is injected in, which means there is low coupling of dependencies. Additionally angular has great separation between views, models, and controllers.

It can be said that WPF or even ASP.NET can use dependency injection or allow you to structure you're code in a MVVM, MVC fashion. However, this is not strictly enforced and new comers often ignore it. Angular compliments .NET here by allowing the server side code to remain small, manageable, and testable. The server side code is leveraged solely for its strengths, which is to do the heavy lifting. The client code is enforcing best practices and also allows for very easy testing through some great unit test frameworks.

With .NET 4.5, through the newly open sourced Web Api 2.0+ and Signalr 2.0+ frameworks, it is extremely easy to build powerful scalable restful Apis. Gone are the days of the configuration heavy WCF. Now to build a RESTful API, you import a library from Nuget and your on your way. This is powerful because you can have a service up and running quickly and it seems to me, in my experience so far, that AngularJs loves RESTful APIs. Binding to Web Api end points in Angular is extremely easy and pain free.

In short We should marry the two because:

AngularJS is definitly the best client side technology I have used so far.
.NET is, arguably, the best server side technology out there.
These two technologie work very well together and never step on each other's toes (Much like the business analyst and software engineer).

Some great resources to get you started.

Learn Angular - Great resource for absolute Angular beginners.
TechEd - Building Rich Apps with AngularJS on ASP.NET - Great Session by John Papa! He has amazing tips for getting productive with Angular and .NET on this video.
Building a Single Page Application with ASP.NET and AngularJS - From Microsoft BUILD 2014
eBook AngularJS for .NET Developers - Althought they never finished the eBook still a great resource.

Using Angular, as a .NET engineer, will allow you to learn the right way to use JavaScript. Which has gained lots and lots of steam in industry since 2009. Likewise leveraging these two technologies together will ensure you are building modern applications, using modern paradigms and tooling. This technology stack allows you to build applications and User Interfaces for many different platforms and Micro Services that are easy to maintain.

I am not saying this type of arrangement is the best architecture for all applications that we build. I am a strong proponent of using the best tools for the job. With that being said, I think this architecture works wonders for the vast majority of the applications we build.

As always if you like the post feel free to subscribe to the mailing list below or follow me on twitter.

Software Engineers should all learn JavaScript.

Louie Bacaj — Sun, 14 Sep 2014 03:46:00 GMT

Yes you read that title right, All software engineers should learn JavaScript.

As Software engineers, we often use JavaScript. But let's face it, JavaScript is like that thing you have had to use to show someone how to consume your API, use your RESTful Service, or to put a front end on your beautiful algorithm. Being primarily a client side tool, with a browser and UI focus, we can't be fully faulted for not really knowing it that well, after all we are busy writing algorithms, services and such. Also, JavaScript hasn’t exactly seemed very stable with a new framework popping up every week; who can keep up? A while back I decided to challenge that thinking and made serious effort to not just use JavaScript in passing, but to learn it properly, while using JavaScript all the time. If you’re reading this, it’s probably because you’re trying to make a change and learn JavaScript yourself.

The case for learning JavaScript

JavaScript is increasingly becoming an indispensable tool, not just for UI work or for us to pretty up the views in our MVC application. JavaScript has become a back end and front end powerhouse that Software Engineers can no longer ignore. It is becoming a huge source of jobs, not just at startups but also at large enterprises. It has solidified its place as the language of new age technologies and software development, much more so than any other language. The way backend software engineers are used to using JavaScript is outdated. Being a back end kind of guy or gal is no longer a valid excuse for not knowing JavaScript.

This isn't your daddy’s JavaScript.

JavaScript today is not the JavaScript of the nineties that we may have used to spruce up our web page. JavaScript now offers a powerful and intuitive set of frameworks and features. Features that, when compared to native UI frameworks, such as WPF, make them look like toys and not like the powerful building blocks they were once considered to be. Through NodeJS JavaScript has features that make building server side APIs extremely easy and fast. Then there are full-blown MVC frameworks, such as AngularJS, EmberJS, and BackboneJS, that are increasingly becoming the default way of building applications. The default way of building, not just web applications but also mobile apps through Cordova and Phonegap. More importantly, JavaScript, through these frameworks and others, is increasingly creeping up on the desktop space and the enterprise through WinJS and other libraries that give native access. All of this is happening at large corporations and not just small startups; just take a look at the massive spike in demand for Angular engineers and the large salaries they seem to command.

Where to Start?

As back end engineers though, especially those of us with Java, C#, and C++ backgrounds, it is a daunting task to get started with JavaScript. We are used to strongly typed compiled code that gives us feedback on a button press. Where do you get started with JavaScript? What frameworks should you pick to learn? What's fastest way to learn it when you already know how to program well? Obviously, the answers to all of this will vary from person to person, but I have some useful resources below that may be able to help guide you. At least the resources below will help reduce the time wasted while getting started. JavaScript commands so many resources online but much of that stuff we can easily skip.

I will share some of my ideas on the best way to learn JavaScript for software engineers. The way that worked best for me and this way you can avoid wasting time digging around and see fast results. Keep in mind these tips assume you have somewhat in depth knowledge of at least one object-oriented language.

Tip 1: Most of us are already fairly familiar with JavaScript, have used it in passing or used it for some of our views when working with MVC frameworks on the server side. The best way is to quickly refresh yourself with the syntax, the command structure. A quick resource to do this, not just for JavaScript but for many other programming languages, is “Learn X in Y minutes”. If you are not that familiar with JavaScript, however, and want something more in depth, the Mozilla foundation has an awesome guide that will show you all of the fundamentals of JavaScript and that can be found here. Once you have familiarized yourself with the syntax your ready for tip number two.
- “Learn X in Y minutes” - For a quick JS recap.
- Mozilla Developer Network Guide - Great in Depth guide.

Tip 2: As back end guys and gals, we should start with what we know best, the back end! The way we do that is to start learning NodeJS. Node is perfect because it’s the same kind of work you do every day in Java, C# or any other OO-Language your familiar with. Node is also perfect because it has some amazing learning material to start with and all of this can be found at Node School. You can solidify the syntax of the language and some of the more advanced features, such as modules and callbacks etc. The beauty of starting with Node is that we can initially focus on building simple APIs and maybe some RESTful services. Node’s npm package management system is easy to use and will have you building useful back end tools in no time and really committing the syntax to your brain. You may disagree with some of the design choices of Node, such as the all-asynchronous IO system and basically it’s asynchronous approach to everything, but remember the point is to learn JavaScript.
- NodeJS - Quick double click installer and provides all the docs!
- Node School - Awesome follow along tutorials (All langugaes and frameworks should have this!).

Tip 3: Then you need to start to ease yourself into some of the more useful frameworks for the front end and app development. After all, the front end and app development is where JavaScript really shines. Regardless of your disagreements on the merits of Node on the backend, it will be hard to disagree with the power of JavaScript on the front end. One of the easier frameworks, I have found, to grasp has been KnockoutJS. Knockout allows for two-way data binding and observables. You can bind knockout to your back end API and see the results immediately. This is powerful and allows you to build most small web apps with ease. For maximum results try consuming the APIs you build with node! Knockout has some good learning material right on their main site and more importantly it is a good way to ease yourself into some of the more powerful full-fledged frameworks, which we will talk about in the next tip.

Tip 4: For modern web apps, and to really leverage the full power of JavaScript, you'll want to learn about some of the full-fledged application frameworks. Much like the MVC frameworks we use with Java and C# things such as ASP.NET MVC or the Apache Struts; JavaScript has full blown MVC frameworks but the crazy part is everything is done in JavaScript, the models, controllers and yes obviously the views all in JavaScript and all on the client side. This is totally diffrent say from ASP.NET MVC where our models and controllers sit on the server and only our views sit on the client. These frameworks allow you to build the entire application in just JavaScript and right on the client side. That may sound silly to us enterprise developers but these frameworks are highly evolved, extremely popular, active and loaded with features. The three major single page application frameworks are Angular, Ember, and Backbone. You can read more about them here and many other places on the web and evaluate them for yourself.
- Angular vs Ember vs Backbone - Great non-biased evaluation of the top 3 frameworks so you can pick the one you like to learn.

Tip 5: Pick a framework and build an app. More importantly everything you build on the backend force yourself to build a client on the front end and leverage one of these application frameworks while doing it. I chose to really focus on Angular simply because of its shear popularity and the fact that it has the backing of Google but any should work. Build apps with your framework of choice and consume your back end APIs. Becoming an expert with one of these frameworks will really help you understand the true power of JavaScript. The good news is that this isn't that different than building an MVC app with the server side languages we are so used to. All of those principals still apply; the difference of course is that you will become a JavaScript expert.
- Learn Angular - If you happen to chose to learn Angular as I did then I suggest Learn Angular as it is a great step by step guide.

JavaScript is here to stay!

JavaScript, whether we like it or not, has become the assembly of the Internet. It is very much the backbone of Internet applications and is encroaching into many other territories. We can close our eyes and try to ignore it, but it is increasingly becoming the language of new age applications. It commands a massive amount of open source projects and many users. This language also is increasingly commanding some of the best jobs in our industry.

The skill set of people who “know” JavaScript varies significantly because of its low barrier to entry. The good news is it seems to me that back end software engineers make excellent JavaScript and new age engineers if they chose to take the time to learn this language and some of its more powerful frameworks.

So what are you waiting for go start learning JavaScript!

(Don't forget to subscribe to the e-mail list if you found this useful.)

Web API 2.0 Service with a Java Client.

Louie Bacaj — Wed, 23 Jul 2014 22:23:00 GMT

Web API RESTful File Server and Java Client.

Writing APIs that are cross-platform and cross-language has not always been an easy thing to do and certainly hasn’t always been light weight stuff. However, with JSON taking over the world thanks to JavaScript, RESTful APIs seem to be emerging as one of the best ways to do this sort of thing. Being resource driven the RESTful API certainly has its limitations, e.g. when streaming lots of data, but for most use cases this type of API is more than adequate. If you don’t believe me then believe Google as they use them all over the place.

Up until recently in .NET, building a RESTful API required lots of work and was anything but light weight. It entailed setting up lots of configuration end points, dealing with service contracts, and setting up interfaces through WCF.

All of that has since changed; Web API was introduced and was recently streamlined into Web API 2.0. This framework is heaven for C#/.NET services developers. It allows you to get a RESTful API in .NET hosted inside IIS or as a Windows Service (Sometimes referred to as self hosted via OWIN) in less than an hour.

I am going to build a simple API that serves up files from the file system using ASP.NET Web API 2.0 and consume that same API via a client in Java. It will look something similar to this.

Web API end points can be consumed via any language that has a HTTP Client implementation, which is basically almost all programming langugaes.

So you may be asking “Why Java?”: The answer is simple in that this is a very common use case. Java is a very popular language and widely used in the enterprise. Since enterprises have lots of services; Java will be one of the most common ways to consume these end points. Although this is not very difficult to do very few examples showing how to do it actually exist. Obviously the most common use cases are consuming Web API via JavaScript and .NET on the client side but Microsoft has plenty of great tutorials on how to do that.

Now let’s get to work!

All of the code is on GitHub so you can follow along.

We will build a Web API Service that will serve up files a from folder on the server side. Normally in a real scenario these files will come from some document store, in some cases a database, or even another API.

Lets add some unit tests as well!

Now we should have a fully functional Web API Service. Visual Studio gave us a free ValuesController which we aren’t going to use but is a nice stub for you to add your functionality to.

Let us rename the auto-created ValueController to FileServerController and delete all of the stub functions created for us in there as we will write our own. Let us also declare the location where we will store our files on the server side.

It is also really great practice to inject in any external objects into our controller rather than instantiate them directly inside. For that we will use the Unity dependency injection framework. Create a constructor for the controller that takes in a single object of type IFileServer as follows:

public class FileServerController : ApiController
{
    private readonly IFileServer _fileServer;

    public FileServerController(IFileServer fileServer)
    {
        _fileServer = fileServer
    }
}

Next let’s create the IFileServer interface by right clicking on the main project and going to "Add new item" and under code adding the interface as follows:

public interface IFileServer
{
    IList GetFiles();

    FileInformation GetFileInfo(int id);

    Stream GetFileData(int id);
}

Our interface now has some very simple functions. Visual Studio may be throwing an error regarding the "FileInformation" object which does not yet exist.

Next let us go to our models and create that. Under the models folder right click and add a new Class and name it FileInformation and it should look like the following:

public class FileInformation
{
    [Key]
    public int Id { get; set; }
    public string Name { get; set; }
    public string Extension { get; set; }
    public string FullFilePath { get; set; }
    public string Description { get; set; }
    public DateTime CreatedTimestamp { get; set; }
    public DateTime UpdatedTimestamp { get; set; }
    public string ContentType { get; set; }
}

Now let us implement the IFileServer interface we created earlier by adding a New Class to our project called FileServer. Normally if we were talking to a database or some document store we may want some more abstraction here by having a Context Class etc. but we will keep things really simple.

Initially our class looks really simple:

public class FileServer : IFileServer
{
    public IList GetFiles()
    {
        throw new NotImplementedException();
    }

    public FileInformation GetFileInfo(int id)
    {
        throw new NotImplementedException();
    }

    public byte[] GetFileData(int id)
    {
        throw new NotImplementedException();
    }
}

Let us add some real functionality now. Such as the location where we will look for the files (Again this would normally not be a hard coded value but either a document store or a setting in our "App.Config or Web.Config").

private static readonly string ServerFolder = "C:\\Temp\\WebApi";

private Dictionary _fileDescriptions;

A constructor that will fill our internal dictionary:

public FileServer()
{
    _fileDesc = new Dictionary();
    FillDictionary();
}

A function which will fill our dictionary called FillDictionary().

private void FillDictionary()
{
    //We are going directly to the file system
    //To fill our internal dictionary but
    //Ideally we would get these somewhere else.

    FileInfo fi;
    int id = 0;

    var files = Directory.GetFiles(ServerFolder).ToList();

    foreach (String file in files)
    {
        fi = new System.IO.FileInfo(file);
        _fileDesc.Add(id,
            new FileInformation
            {
                Id = id,
                Name = Path.GetFileName(file),
                Extension = fi.Extension,
                FullFilePath = fi.FullName,
                Description = "temp image",
                ContentType = "application/png",
                CreatedTimestamp = fi.CreationTime,
                UpdatedTimestamp = fi.LastWriteTime
                });
            id++;
        }
    }

Next up let us implement our functions from our interface.

Getting the file list is really easy since all the files will be inside of our dictionary when we create the FileServer.

public IList GetFiles()
{
    return _fileDesc.Values.ToList();
}

The GetFileInfo() function is also very easy to implement.

public FileInformation GetFileInfo(int id)
{
    FileInformation fileInfo;
    if (_fileDesc.TryGetValue(id, out fileInfo))
        return fileInfo;
    else
        return null;
}

Getting the File Data is also very straight forward and using standard .NET APIs works great for this.

public Stream GetFileData(int id)
{
    FileInformation fileInfo;
    bool ok = _fileDesc.TryGetValue(id, out fileInfo);

    if (ok)
        return new FileStream(fileInfo.FullFilePath, FileMode.Open);
    else
        return null; 
}

When we build or run our solution we may get an error about our controller not being accessible. This is because we are injecting in our FileServer. We will need to create some boilerplate code which basically allows Dependency Injection to work with Web API.

Again if we wanted to simply remove the dependency injection we would change our controllers constructor to the following:

public FileServerController()
{
    _fileServer = new FileServer();
}

(If you skipped the Dependency Injection by creating your object inline then you can skip this section and go right to the the Web API Controller Service)

Web API Dependency Injection

If you chose not to skip then we will do things the right way here and inject our FileServer. We will need to add Unity via Nuget.

Next we will add a required class which is basically right from Microsoft and allows you to use Unity as your DI framework. I added this to a folder called DI but you can add it in the main project and that will work fine.

public class UnityIoC : IDependencyResolver
{
    protected IUnityContainer container;

    public UnityIoC(IUnityContainer container)
    {
        if (container == null)
        {
            throw new ArgumentNullException("container");
        }
        this.container = container;
    }


    public object GetService(Type serviceType)
    {
        if (!container.IsRegistered(serviceType))
        {

            if (serviceType.IsAbstract || serviceType.IsInterface)
            {
                return null;
            }
        }
        return container.Resolve(serviceType);
    }

    public IEnumerable GetServices(Type serviceType)
    {
        try
        {
            return container.ResolveAll(serviceType);
        }
        catch (ResolutionFailedException)
        {
            return new List();
        }
    }

    public IDependencyScope BeginScope()
    {
        var child = container.CreateChildContainer();
    return new UnityIoC(child);
    }

    public void Dispose()
    {
        container.Dispose();
    }
}

After adding that class we then need to go to our App_Start folder and find the WebApiConfig class. This class is where our Web API routes are configured and where our API comes to life.

In here we need add to the configuration object our DependencyResolver as follows:

var container = new UnityContainer();

container.RegisterType(new HierarchicalLifetimeManager());

config.DependencyResolver = new UnityIoC(container);

The whole WebApiConfig class looks as follows:

public static class WebApiConfig
{
    public static void Register(HttpConfiguration config)
    {
        // Web API configuration and services

        // Web API routes
        config.MapHttpAttributeRoutes();

        var container = new UnityContainer();
        container.RegisterType(new HierarchicalLifetimeManager());
        config.DependencyResolver = new UnityIoC(container);

        config.Routes.MapHttpRoute(
            name: "DefaultApi",
            routeTemplate: "api/{controller}/{id}",
            defaults: new { id = RouteParameter.Optional }
        );
    }
}

To find out more on Web API Dependency Injection and how it works go to Microsoft at Dependency Injection for Web API Controllers

The End Points.

Let us now wire up our application logic with our end points. We go back to our FileServerController class and add our end point functions.

[RoutePrefix("fileserver")]
public class FileServerController : ApiController
{
    private readonly IFileServer _fileServer;

    public FileServerController(IFileServer fileServer)
    {
        _fileServer = fileServer;
    }

    [Route("allfiles")]
    [HttpGet]
    public IEnumerable GetAllFiles()
    {
        return null;
    }

    [Route("download/{id}")]
    [HttpGet]
    public HttpResponseMessage DownloadFile(int id)
    {
        return null;
    }
}

A few things to point out in the above code. The routes to our API from the URL can be fixed via the WebApiConfig or they can be hard coded via data annotation as done above. The above is usually not a good idea for a really large service but for something small it helps highlight where you will land by following the route.

Next let us wire up our application logic which is in our FileServer class to our end points.

We can see that to get all files from our file server will be really easy.

[Route("allfiles")]
[HttpGet]
public IEnumerable GetAllFiles()
{
    return _fileServer.GetFiles();
}

To actually download one of the files through our API is also fairly straight forward as shown below.

[Route("downloadfile/{id}")]
[HttpGet]
public HttpResponseMessage DownloadFile(int id)
{
    var fileInformation = _fileServer.GetFileInfo(id);

    var path = fileInformation.Name;
    var result = new HttpResponseMessage(HttpStatusCode.OK);
    var stream = _fileServer.GetFileData(id);
    result.Content = new StreamContent(stream);
    result.Content.Headers.ContentType = new MediaTypeHeaderValue(fileInformation.ContentType);

    return result;
}

We are creating a HttpResponseMessage and we are embedding our stream of data from the file system to that.

We are now officially done with the Web API Service! If we right click and run that under debug we should be able to see our APIs.

Java Client for the Web Api Service.

Next up let us create our Java project using Eclipse or your favorite Java IDE. I am using Eclipse but any Java IDE will be just fine.

Let’s add a few JAR files that will allow us to communicate with our Web API RESTful Service. These JARs are fairly standard one is from Google called GSON and the others are from Apache. Since we built a standard RESTful service we simply need an HTTP client to consume the end points and we use Google GSON to map the JSON that is returned to an internal Object (Or Bean as they say in Java).

Those JARs can be found by searching for Google GSON and Apache HTTP Client to get the latest or follow the below links.

Obviosley you can use Maven or any other mechanism to get these as well.

Let us add a new main class that will consume the Web API endpoints.

public class JavaWebApiApp {

    public static void main(String[] args) {
       // TODO Auto-generated method stub
    }
}

We create our bean which really maps one to one with our Object in C#. Except I will map the DateTime items to Strings in Java.

public class FileInformation {

private int Id; 
private String Name;
private String Extension;
private String FullFilePath;
private String Description;
private String CreatedTimestamp;
private String UpdatedTimestamp;
private String ContentType;

//Getters setters etc.
//...
}

Next up let us write the code in Java that will consume the end point and grab the objects from our RESTful API.

public static ArrayList httpGet(String url, String function) {

    ArrayList dataFromService = new ArrayList();
    try (CloseableHttpClient httpClient = HttpClientBuilder.create().build()) {
        HttpGet request = new HttpGet(url+function);   
        request.addHeader("content-type", "application/json");
        HttpResponse result = httpClient.execute(request);
        String json = EntityUtils.toString(result.getEntity(), "UTF-8");  

        com.google.gson.Gson gson = new com.google.gson.Gson();                       
        FileInformation[] response = gson.fromJson(json, FileInformation[].class);

        System.out.println(response.length);   
        for(FileInformation file : response)
        {
            dataFromService.add(file);
             System.out.println(file.toString());
        }

    } catch (IOException ex) {
    }
    return dataFromService;
}

What we have there is just some basic use of the Apache HTTP Client. We are going to the end point and consuming the returned JSON and binding it to our Bean using Google GSON.

If we call that from our main function:

public static void main(String[] args){

    String url = "http://localhost:53667/fileserver/";

    httpGet(url, "getallfiles");
}

We will now get the following:

Next let us add the ability to download the file.

Add the following to our main:

OutputStream downloadedDoc0 = httpDownloadFile(url,"downloadfile",0);
System.out.println(downloadedDoc0.toString());

Then the following code to download the stream:

public static OutputStream httpDownloadFile(String url, String function, int fileId) 
{

    ByteArrayOutputStream byteOutPutStream = new ByteArrayOutputStream();

    try (CloseableHttpClient httpClient = HttpClientBuilder.create().build()) 
    {
        HttpGet httpRequest = new HttpGet(url+function+"/"+fileId);
        HttpResponse result = httpClient.execute(httpRequest);
        HttpEntity resEntity = result.getEntity(); 
        resEntity.writeTo(byteOutPutStream);

        EntityUtils.consume(resEntity);
   }
   catch (Exception e) 
   {
        System.out.println(e);
   }

    return byteOutPutStream;
}

If you get all of that, that means it worked and you have the data in your Java app!

That is it! We now know how to make a Web API Service and then consume that via Java!

Lets see what the finished API Service looks like.

If you like these tutorials feel free to subscribe.

A lightweight shopping cart web application in ASP.NET MVC 5.

Louie Bacaj — Fri, 20 Jun 2014 15:41:00 GMT

Recently I decided to build a lightweight shopping cart web app in ASP.NET MVC 5 so that it can be used by local businesses to take orders online.

The app takes orders online and sends out an email to the shops owner. The email contains all of the order information and the business owner can charge it using their in store credit card management system.

It is very light weight, responsive, secure, and open source and it is built on top of some really great technologies, all of which are open source as well.

The source code can be found on GitHub here.

The problem.

I have a lot of family and friends in the restaurant industry and others that own small businesses and they are all facing the same type of dilemma:

How to take advantage of the internet and the explosive growth of mobile apps to expand their business.

They want to allow their customers to place an order online. They want to charge it themselves and ship or deliver the items themselves. They don’t want any fancy system, a CMS, or anything with too much complexity and to be frank they can’t afford to maintain any of that.

They want simplicity. Once the app is deployed and running they can maintain it with ease. I set out to build an extremely lightweight framework that I could reuse across all of these different small businesses and then I decided to give it away and open source it.

Most of these small businesses have static websites that show exactly what they are selling but have no way of actually selling anything online without adding complexity or cost.

Don’t get me wrong apps like seamless and grub hub are great. They allow restaurants to sell online and have wide reach but they also charge restaurants anywhere between 10% to 20% of the order price. Not to mention other types of shops don’t even have services like these available to them.

By providing a fully finished secure web app, that just requires some configuration and deployment to your favorite hosting site, this framework aims to give the power back to the local shop owners.

Why build another shopping cart?

While there are many great shopping carts out there, done in any programming language used to build web applications, very few of them solve this problem in a simple easy to maintain secure manner.

Most of these existing carts are bloated with lots of features which can really mean lots of complexity. Take for example nopCommerce, it took me longer to setup items and configure a shop in there then it took me to write my own framework that I could reuse.

Don’t get me wrong features are nice but for the local shops and small business owner I think just a clean way to take orders on their website is more than enough.

Now for my favorite part.

The app uses a lot of exciting technologies that make development extremely easy and fast. (I build this on the side after work and in less than two weeks from start to its current state.)

The Technology Stack.

ASP.NET MVC 5 and .NET 4.5 at its core written in C#. I took advantage of some of the asynchronous programming features of .NET 4.5, I would like to stress some because I have more work to do on this one. The app is extremely fast and very responsive and in all honesty was a dream to write because these technologies are so nice to work with.
HTML 5 – Front end development is extremely easy with a framework like bootstrap. I simply took a free open source theme from http://bootswatch.com and had a beautiful responsive UI that works great on mobile and desktop within a few minutes.
Entity Framework 6 code first migration for all of the database work. This makes it extremely fast and easy to deploy and working with Entity 6 is so much fun. Allows for easy asynchronous write backs and saves to the database without having to worry about writing any SQL. (On a side note I once told a colleague of mine back at SunGard that I hated writing SQL because it was so 1960-ish, I didn’t mean that I dont really thik SQL is that bad.)
SQL Server Express database, although entity makes it easy to use any database behind this.
Identity 2.0 for the security aspects, Microsoft has some really great template to get you up and running with a super secure site within an hour. This allows for a great authentication and authorization system that supports external logins from Google and Facebook and easily allows for roles.
Mailgun for the emailing system and they have a great free 1000 email program that these small businesses will generally never go over.
Windows Azure for hosting but in reality any ASP.NET hosting provider will do.
D3.js for some analytics and charting so shop owners can track how their online business is doing. I should note that this isn’t fully implemented as I was simply learning D3 while doing this. More will be done on this one in the future for sure as D3 is one of the best UI technologies I have come across.
There are also lots of other technologies such as
- Visual Studio
- TFS Online
- Github
- jQuery
Oh and I would really like to shout out AppVeyor they are extremely easy to use all in one continuous integration system that automatically builds and tests your app right off of github. It is truly a joy to work with and don’t just take my word for it go see what one of my favoret developers has to say about it, Hanselman's Blog.

Finally the app is loosely based off the ASP.NET MVC 3 Music Store from Microsoft, MVC Music Store, but unlike that app this is much more full featured and suitable for the real world. That app is a great way to learn how to build a shopping cart though and my thanks’ goes to the developer, Jon Galloway.

ASP.NET MVC and Google Street View / Static Maps API's.

Louie Bacaj — Tue, 08 Apr 2014 15:40:00 GMT

During the building of Realintory, a Cloud based SaaS application to manage real estate inventory, I ran into a situation where I needed to pull a static map of a location and a street view image of it from the Google Maps API.

Adding Google Maps API integration to an ASP.NET MVC app is pretty simple and this is a sample on how to do it in C# using ASP.NET MVC 5. Now this sample is quick and in the live code base it actually pulls these images asynchronously and stores them into Azure Blobs but you may just want to display them in your view.

On the View

The code is fairly straight forward here, the address does not need to be hard coded you can retrieve it from somewhere else of your choosing on the view, like an input control.

This is just making the call out to the controller and calling the ShowGoogleStreetViewPhoto(String googleStreeViewAddress) and ShowGoogleStaticMap(String googleStaticMapAddress).

On the Controller

We need two functions on the controller to return a FileResult back to the View, basically each will return an image file.

public FileResult ShowGoogleStreetViewPhoto(
String googleStreeViewAddress)
{
     //Size up to 640-640 on the free version
     Task result =   
     GetStreetViewImage(googleStreeViewAddress,"500x500");

    //Quick check to see if we have an image
    if (result == null)
           return null;

    var file = result.Result;
    return File(file, "image/jpeg");
}


public FileResult ShowGoogleStaticMap(
String googleStaticMapAddress)
{
    //Size up to 640-640 on the free version
    Task result =
    GetStaticMap(googleStaticMapAddress, "500x500");

    //Quick check to see if we have an image
    if (result == null)
           return null;

    var file = result.Result;
    return File(file, "image/jpeg");
}

You may have noticed that they each call two other functions on the controller, or wherever you want to place them, to grab the actual images from google.

These functions are very similar you may only want one of them if you want to just return an image or maybe just a static map.

public async Task GetStreetViewImage(
string location, string size)
{
    if (string.IsNullOrEmpty(location))
    {
        return null;
    }

    try
    {
        var streetViewApiUrl = "http://maps.googleapis.com/maps/api/streetview?sensor=false&";

        var formattedImageUrl = string.Format("{0}&size={1}", string.Format("{0}&location={1}", streetViewApiUrl, location), size);
        var httpClient = new HttpClient();

        var imageTask = httpClient.GetAsync(formattedImageUrl);
        HttpResponseMessage response = imageTask.Result;
        response.EnsureSuccessStatusCode();
        await response.Content.LoadIntoBufferAsync();

        return await response.Content.ReadAsByteArrayAsync();

    }
    catch
    {
        return null;
    }
}


public async Task GetStaticMap(
string location, string size)
{

    if (string.IsNullOrEmpty(location))
    {
        return null;
    }

    try
    {
        var stasticMapApiUrl =   "http://maps.googleapis.com/maps/api/staticmap?&markers=color:navy%7Clabel:R%7C62.107733," + location + "&zoom=12&maptype=hybrid&sensor=false";

        var formattedImageUrl = string.Format("{0}&size={1}", string.Format("{0}¢er={1}", stasticMapApiUrl, location), size);
        var httpClient = new HttpClient();

        var imageTask = httpClient.GetAsync(formattedImageUrl);
        HttpResponseMessage response = imageTask.Result;
        response.EnsureSuccessStatusCode();
        await response.Content.LoadIntoBufferAsync();

        return await response.Content.ReadAsByteArrayAsync();

    }
    catch
    {
        return null;
    }
}

What does it all look like?

Configuring the API Calls

To get the most out of your images take a look at Google's Documentation on setting up the URL parameters used in the code above.

As always please feel free to subscribe. I will be posting code snippets while building Realintory and maybe some of that is useful to someone out there.

Building Realintory - Part 1

Louie Bacaj — Tue, 01 Apr 2014 16:32:00 GMT

Justin Jackson, from JustinJackson.ca, always says that the best applications are born out of solving a problem for a particular group of people. If that is the case, then Realintory should be no exception.

The Problem the Real Estate Industry is Facing:

Currently, managing inventory is done by hand, especially for rental properties. There is a big problem with double listings by the same office or real estate agency. Until now, agents and brokers have managed and tracked their listings via spreadsheets and paper, which is very time consuming as well as unreliable. Another problem is that one cannot extract any useful analytics from paper and spread sheets to run any serious analysis that would promote the business. My good friend Andrew Haxhiu, a real estate agent for many years, has been working closely with us and providing valuable feedback on the current challenges of an agent during the entire SDLC process for Realintory.

How Will this Problem be Solved with Realintory:

Realintory is a solution for real estate agents and brokers to manage their inventory from any device. We intend to make it the most intuitive way for real estate professionals to manage their rental properties as well as properties for sale.

Additionally, since the application is cloud based it has the added value of allowing seamless integration with services like Zillow, Trulia, and the MLS. What that means is that real estate agents will only need to enter a listing once and push it out to all of those services right from Realintory. In addition to saving time during the tracking it is also a great resource for looking up inventory and doing comparisons from anywhere and any device.

Another great value that brokers and agents will get from our application will be in the form of valuable analytics on the homes they have in their inventory. The Realintory Analytics tools will empower the agents to compare home trends across different periods of times as well as zones therefore increasing the transparency for both buyers and sellers that they serve which in turn increases the value to their customers.

The Team:

The team right now consists of me, a full stack engineer with an M.S. in Computer Science that most recently worked at a Fortune 500 Software and IT Services Company. I have helped scale out multi-million dollar projects to Tier 1 financial institutions and have worked with many different technologies in the process. My brother Anton Bacaj, who is a phenomenal UI designer and software engineer, he is currently enrolled in a master’s program for Computer Science and has helped build many web sites and web apps as a consultant under Luce-Designs. Websites such as AlexPerlov.com, EzGrillNYC.com, SalvatoresOfSoho.com, and other apps such as the Fitture Windows Phone 8 app. Our realtor expert Andrew Haxhiu who has worked in the industry for over 5 years and owns and manages several commercial properties. We are enticing some other engineers to come onboard in the near future. We had a lot of architecture and platform decisions to make and do it quickly.

High Level Architecture:

Realintory, under the hood, is a cloud based Software as a Service (SaaS) solution, which means it will have:

Services on the cloud (API based)
A web interface
Mobile apps (Android and iOS initially)

The benefits of the architecture:

This cloud implementation will seamlessly allow agents to enter listings and take photos of the properties right from the field on their iPad or Android phone. Additionally this architecture will allow the application to keep the entire office or agency in sync and constantly provide valuable analytics. Having to enter a listing only once in our app and pushing it out to all the major real estate applications, like Zillow and Trulia, will save agents valuable time.

Tools and Technologies:

For Realintory We decided to go with:

Windows Azure - For our infrastructure and to host our services and application.
- ASP.NET MVC 5 - Server side and web application development.
- Web API 2.0 - Endpoints for our mobile apps.
- Xamarin - Android and iOS app development.
- HTML 5 - Front end for the web app and simultaneously leverage some of that for our mobile apps as well.
- Entity 6 – Code first migration for the database side
- Lots of others like Visual Studio, TFS Online, Windows Azure Blobs and Tables, SQL Server, jQuert, Bootstrap and many more.

Reasoning behind these technologies:

We landed on these technologies because our day to day stack is C#/.NET and Java in our full time jobs and because we wanted to leverage that to build this in an iterative approach. Mainly get something out there as soon as possible. Having successfully used Azure before to build some web apps and mobile services, it seemed like a no brainer for a SaaS application like this to be built in that way. We will look to convert certain parts to other technologies, like Node.js, in the future, especially some of the collaboration aspects we plan to add to it down the road.

Ultimately, we are looking to build this on solid foundations, as well as in an agile rapid development manner. Things like security are handled for us via Identity, while caching and document based storage are easily done using Windows Azure.

P.S. Don’t forget to subscribe to the blog newsletter if your interested in my blog posts or want to know more about Realintory as we develop it.

Beware of recruiters.

Louie Bacaj — Fri, 28 Mar 2014 18:38:00 GMT

I would like to start by pointing out that I have worked with many fantastic recruiters, whom have given me great advice and career tips. With that being said, I may continue.

The tech industry is so hot right now; anyone with a decent LinkedIn profile is going to get a few requests a week from recruiters about possible opportunities. Most InMail’s will go ignored, as developers are usually busy working and writing code. Occasionally though, one or two great opportunities may land in your mail box and you may get the urge to respond. If this isn’t happening for you then you need to work on your profile, in the very least so you can see what’s out there.

Alternatively, If you have tried something gutsy, like putting your profile on Dice, you are basically, as a developer in 2014, guaranteed to get about 10 calls a week, sometimes as many as 30+. I know this because I have the voicemails to prove it.

Most of that of course will be noise and it could be just recruiters matching words on your resume to those on the job description. It is up to you to examine each opportunity and decide which is right for you.

Recruiters will more than likely bait you by talking about how a move to company X will be

“phenomenal for your career”

and how the salary and bonus will be

"significantly better".

They will latch onto what you don’t like about your current role and push that this new place is different. Fair enough it is their job to entice you to move, they make commission if you do take the job.

The issues arise when they tell you things like:

“The more money you get, the more money I get.”

“I will negotiate vigorously on your behalf.”

These things seem like good deals for you both. Most of the time though, they do not negotiate vigorously. Instead, recruiters try to land on a salary that will not scare the potential employer. Aside from having close relationships with these employers, recruiters would rather take a lower offer than no offer at all.

It is also very important that they appear to be, both, making you more money and saving that employer money. However, these two goals contradict one another. How can they maximize your salary, while saving money for the employer?

When a potential employer views your current salary, they are usually not going to want to give you more than a 20% increase. This is the market norm, unless of course that particular employer is really generious. In fact, as you can imagine, the less money they offer you and get you to take the role, the better it is for them.

A lot of time’s though, recruiters tend to make empty promises by saying things like:

“Don’t worry we can easily get you 25% raise”,

Things like that would make anyone consider changing jobs. But in the end, the conversation changes to:

“Look a 10% raise is normal in this market”.

Don’t believe any of that.

I have worked with recruiters that changed the conversation from “a 25% raise is very possible with this employer” to a “lateral move” being a great move for me and my career because of “X and Y”. They say things like that because they want to manage expectations and some will simply so they can get a commission.

Clearly that is not in your best interests. Set a realistic target by doing research of how much you should get paid. From there, push the recruiter to get you to where you want to be.

A few great places to research where you should be are:

Of course there are many other places as well, but I have found those two places to be particularly useful. In this hot market, unless your salary is way above market value or you really hate your current job, you shouldn’t consider any job below a 20% increase.

A few key points:

Salaries are a personal thing that people don’t like to usually talk about, so it’s important that you take the initiative to do your own research, and find out what your value really is. Don’t rely on recruiters for this, not that they aren’t nice people but that they will have their own agenda.
Whatever you decide to do, don’t let a recruiter bully you into taking a job you don’t really want to take especially if your gut is saying you should get more.
Recruiters will be pushy. Use the recruiter to your advantage and make clear what you’re looking for, so they can work for you and not against you.

P.S. Despite all I have said it is important to keep in mind that money isn’t everything. If you stumble upon a role that excites your or would allow you to further your career and expand your knowledge, rather than just your bank account, you should use your good judgment when considering it.

Please subscribe to get updates about new posts.

Perseverance and Burning out.

Louie Bacaj — Mon, 17 Mar 2014 17:59:00 GMT

Perseverance, or grit, is arguably just as important as intelligence, if not more. But don’t just take my word for it, read Eric Schmidt’s comments, made recently, on the value of education.

Google's Eric Schmidt on criticts who say college isnt worth it, theyre just wrong.

I recently wrote a blog post about the value of doing things, basically just rolling up your sleeves and getting great at what you want to be great at by just persevering. It is also, apparently, fact that people who persevere make more money than people who are just intelligent but give up quickly; the CEO of google was pointing to the fact that our universities need to do a better job of building up grit in their students.

I strongly belive that persevering and burning out are two totally different things. Recently there have been a few blog posts on hacker news that have gone viral, speaking about programmers going insane by burning themselves out. The body and the mind can only take so much.

I knew a programmer that went completely insane

As proof of this, there is a sport known as cross-fit. Cross-fit is basically all about doing exercise until you burn out, until your body cannot take anymore. There have been countless reports of people injuring themselves, tearing ligaments, breaking bones, and in some extreme cases dying. Extreme programming is no different than this; it is basically pushing your mind until breaks down.

Personally, I don’t believe that is healthy. When you work out until you burn out, your body no longer stimulates muscle growth. In fact, it begins to consume muscle mass, essentially eating itself. Try running for hours at a time, you will find your body will initially burn lots of fat but after a certain point, you will lose muscle mass at a much higher rate. The brain, I think, is no different.

Perseverance is about doing something continuously and not giving up, that doesn’t mean in one sitting. Grit is really about not giving up after failure. To properly leverage both and get great at something, you have to do that thing continuously, remembering that life is a marathon not a sprint. There is a threshold, your body and mind will get tired and it will give you signs, you should not cross that threshold as the value you are getting is significantly decreased. Your software will have bugs if you develop tired, side effects and unintended consequences will undoubtedly enter the process. Similarly with working out a 45 minute to 1 hour session in a day is all that is required. You do that for years and you see your health improve; Similarly you write code for an appropriate amount of time and you will see that skills improve as well.

Burning out will not bring the results you want faster, only continuously doing in a healthy way will bring real long lasting results.

In the spirit of doing.

Louie Bacaj — Sun, 02 Mar 2014 22:51:00 GMT

Speaking from personal experience, although I’m sure a lot of you can relate, in life, it seems like some people are just terrific at doing certain things. To outsiders looking in, it seems as though these people were born skilled. At some point or another, we all have probably struggled to comprehend why we just aren’t as good as the next guy or gal.

It took me a while to learn this, but for the rest of us, who are not born excelling at something we would love to be great at, we don’t have to watch in awe and just wish we could do those thing. We need to start doing, and continue doing in order to get good at the things we want to be good at.

It took 5 years of working out, going to the gym 4 to 5 days a week, to really lift some serious weight. It took getting a master’s degree in computer science and constantly trying to implement my own data structures and utilize different algorithms to really understand. I did this by rolling my own data structures and algorithms and forcing myself not to use what’s in the library. I didn’t do this because my implementations were better, in fact they were probably significantly worse. I wouldn’t recommend doing that for any serious projects that are going to go live as it’s unsafe, but boy is it a great way to learn. Much like lifting weights I did it to reinforce muscle memory to truly understand big O notation and get good at writing code. Repetition of challenging tasks can be difficult but it builds grit, muscle, and intelligence.

See, the problem is that doing stuff is hard and most people give up after a few failed attempts. People want immediate results. If they don’t see results after 3 months of going to the gym they stop working out. If they can’t understand how red-black self-balancing binary tree are implemented then they will just give up. If they can’t see education giving them an immediate high paying job as soon as they graduate then they drop out. Most people will quit, they don’t understand that doing the process is the results. You have to constantly improve yourself to be great at the things you want to be great at. When you do things all the time you are guaranteed to see results, that process of doing those things will be your results but the trick is to never stop.

I decided to write on this blog because I want to get good at writing. I go to the gym because I want to be healthy and feel strong. I code everyday whether it’s for work, school, or a personal project because I want to be great at it.

So in short if you’re not that good at something but would really like to be I suggest the following:

You just have to start doing and do it over and over again. You really can’t stop improving and you have to do this because you were not born great at it but you want to be great at it.

P.S. I know we all have qualities that others admire, things that come so naturally, but this post isn’t about that it’s about the things you want to be great at but just aren’t yet.