Phelio Gnomi

My words from my mind

Agile – From a developer’s eyes

What does it mean to be Agile? There are 3 common answers that I’ve summarised:

  1. It’s a corporate bullshit.
  2. Everyone interprets it differently, whatever works for you.
  3. You’re doing it wrong.

Somewhere between those lines, there’s a subset of people who truly believe that Agile is helping them so much in their daily life. And I’m one of them.

I’m afraid I might have over-simplified the common sentiment of Agile in the tech community. However, what I’ve loved so much about this community are the people that have shown me how great Agile can be, not by telling me to do what was told, but by bringing me together in their journey and seeing the values for myself, and structuring frameworks and environments where I get to help build a better culture.

And that’s one of the value in the Agile Manifesto which values “Individuals and interactions over processes and tools”. We values individuals as human beings and trustworthy and values the interactions between them more than telling them to follow the sets of rules or using a specific tool for conveniences of the business.

There are many Agile writers out there that would be more comprehensive and most certainly helpful. But I’ve not seen much that doesn’t mention “delivery” or “customers”. As a developer who’s worked in various teams claiming to do Agile for over 5 years, and if you are also a developer, you would agree that customer/clients/consumers are rarely a part of our job. However, we often shape our processes around satisfying these imaginary customers that we won’t ever met. Feedbacks that got swallowed by product team because they don’t want to bore us with details, showcases for the sake of showcasing, estimating for the sake of estimation.

I’m not arguing that the customers are not the highest priority, they are. What I’m trying to say is that we’ve done so well in satisfying the customers we sometimes forget about the people behind it.

People is my passion. Human mind intrigues me and it’s my daily amusement to learn about myself, others and the interactions between. In this series, I’d like to share my stories from a developer’s point of view on how Agile has helped me as a developer and making my job less miserable. They’ll be focusing around people and not just developers but from a developer’s point of view. Stay tuned for the next one.

Advertisements

On workspace design and work culture

Despite being labeled as “hard to change”, culture is actually a very fragile thing. You’d often hear people say that it’s hard to build a culture. But also true that it’s difficult to maintain a good culture. So which is true?. I also think that workspace design has a subtle but important effects on shaping culture at work. let’s explore some of the examples of the modern workspace design and how they can negatively impact the culture.

Hot desks

Hot desking is when the employees are not designated with a fixed seat.

Boosting competitiveness and selfishness

Where one sits depends on how early they get there every day. Most of the time, their seat is reserved for them, their team mates know where their preferences are and I’m surprised to find that most of the time I actually find my seat to be empty.

However, when challenged with expansion or on team movements, the seating arrangement will change and slowly become more settled again.

It sounds like a good approach, but what culture are we really building here? The obvious good one is flexibility. Also the flock mentality that dictate that the people who works together will move in a similar fashion.

On the downside, it’s also building competitiveness. Not every desk is the same. Some has better monitors, some has better ergonomic. Often when complaining about not getting a good seat, the answer is “well, you should have came earlier”. But is that the culture that we wanna have in a productive environment?. There will always be someone that’s last to come in and will end up with the less idea desk. That person, even it’s not you, is someone who is a valuable member of the company. Having the setup that build a culture that put others in a less ideal position for one’s benefit is simply toxic.

Stripping you off you

Yeah sure, it sounds like a good thing, having everyone in the same behaviour. No conflict, no war.

No, it’s not that simple. We as an individuals are innately different and there’s that natural instinct to be able to tell one from another. Having a hot seat system means there’s no way to customise one’s desk. Every desk looks the same. There’s no way to fashion a photo of the loved ones or having fancy productivity boosters like custom keyboards or funny memes without the price of packing and unpacking them everyday.

Culturally, it’s creating a bunch of robots without creativity. Not allowed to customise their desk leads to the feeling that we are not meant to change anything, that self expression is not encouraged.

Employee engagements

Familiarity can be a powerful tool. People are drawn toward nostalgia for example, that forgotten familiar thing of the past. Hot desking doesn’t allow that to happen. Before one has settled in they have to move away. Not just the desk they are siting on, but the surrounding environment like the people next to them, the location from the toilet, the morning walk from the front door.

When a simple daily task is not familiar everyday, it can feel unsafe, like sleeping in a hotel room or being at a friend’s house for the first time. It’s difficult to feel at home when one is not familiar with it or to be able to build a routine.

Culturally, it’s building detachment. It doesn’t allow an employee to be fully part of the team. And when we don’t feel at home, we don’t feel safe.

Open Plan

Open plan office is when an office has no barriers between employees.

Diversity buster

The most common complained disadvantage of open plan offices are the noise. It’s common that conversation will pop up easily anywhere. It could be a good thing to encourage communication.

However, for some individuals, this form of communication is less favourable for them. Communication shouldn’t be forced but should be natural. More often, the lack of the safe space is the hindrance to communicate. Open plan offices doesn’t solve that problem.

Culturally, it’s discouraging diversity on communication preference. Everyone is expected to be loud and clear, or to be interacting frequently.

Ignoring trust

Supervision is one of the benefit of having an open plan workspace. Having each other seeing what one is doing all the time is meant to be more productive because it reduces distraction and personal stuff.

But it also destroys trust, or neglect the need of one. You don’t need trust if you can see them all the time.

Team work when it’s not needed

It’s nice that most workplaces have great and kind people who are happy to help when needed. Sometimes collaboration will only be effective if everyone is in the same space.

However, even in an open plan, one can pass their day without talking to anyway but still be productive. Friendship is often forged from collaborations and meetings which often happen inside a closed space or outside of the office.

It’s also annoying to disrupt someone when they have their headphones on or when they’re concentrating on something. And sometimes people are actually annoyed by one’s constant nagging for help and most people don’t hide it well.

Culturally, it’s building insensitive. It’s encouraged that one shouldn’t be afraid to walk to another person and disturb them. And not just the person we are interacting but other people around us that could be distracted by things that we do.

I guess what I’m trying to say here is that we should be more aware of some negative impacts everything can bring in a culture. Most of these impacts are not a showstopper. But it’s important to acknowledge them and try to mitigate them when possible.

Debugging PHP the hard way

Having to maintain a massive Object Oriented web application written in PHP, it’s amazing how many times I’ll have to use the same functions over and over again in multiple different places. I’m listing them down here before I forget.

error_log($message)

The most used one, and the only way I can print anything right now in the application I’m working on without screwing up everything. It prints out the message into the application error log, which I then tail -f and keep it running in one monitor. Way to go multiple monitors.

get_class($object)

How do I live without this. Even with the power of IDE ( I’m using NetBeans ), it’s frustrating sometimes to find out what the class of the object actually is. Especially in “well designed” object oriented application full of decorator and factories, it’s easy to get lost. get_class() produce the exact class of the object. Can’t live without this function.

var_export($object, $return=false)

var_dump() used to be my ultimate savior, but with the limitation on the printing into the web without breaking anything, I’ll have to print everything to the log file. var_export() does exactly that when the “return” flag is set to true. Although most of the times it ended up with recursion error, it’s still pretty useful.

Exception()->getTraceAsString()

Got to find the way to trace where something started. Pretty straight forward when combined with var_export() and error_log() as mentioned in one of the brilliant StackOverflow answer.

$e = new Exception;

error_log(var_export($e->getTraceAsString(), true));

What are your favorite debugging snippets in PHP?

Sharing objects between Modules in AngularJs

It turns out that sharing objects between angular modules is as simple as injecting a module into another module. Say myApp has some controllers and providers or other objects and myApp2 want to use some of those controllers. When creating myApp2, inject myApp into it and myApp2 now has access to all the objects in myApp to the extend that the HTML can use the controller from myApp without any extra codes.

// Common library of some sort
app = angular.module("myApp", []);

app.value('someVal', 'This is a val');

app.controller("controller1", ['$scope','someVal', function($scope, someVal){
$scope.someVal = someVal;
}]);

// My actual module
app2 = angular.module("myApp2", ["myApp"]);

app2.controller("searchController", ['$scope', function($scope){
// ... some controller's codes
}]);
<body ng-app='myApp2'>
<div ng-controller='controller1'>
{{someVal}}
</div>
</body>

Note that the HTML is using ‘controller1’ which is from ‘myApp’. I’ve built an app in AngularJs and going to create another one using the similar format. It also means that most of the codes are going to be shared. I’m glad that I don’t have to do much refactoring.

Methods of getting direct feedback from Servers

With the web technology advancing so rapidly and information is getting bigger and flowing faster than ever, many web applications nowadays can’t live without constantly checking for new data from the servers.

The most basic form of a websites deals with HTTP request from the webpage to the server. The user send a request in an url form and the server response with the content that was requested. End of story.

Say that you are viewing a page that tell you how many times a jumping sheep has been jumping and it is jumping in average of once every 1 – 10 minutes, you might want to refresh your page every few minutes if it has been jumping or not. But with today’s technology, there should be a way that we get notified whenever the sheep jumps.

The main challenges of getting a direct feedback is that web application usually support a huge amount of users and they could be located anywhere in the world. It is simpler to think that the server just give the information whenever it’s requested. Instead of being busy trying to send updates to every connected clients (if they are connected). However, let’s explore the possible implementations.

Auto refresh

If you’re building a website in the 1990s, this is probably a very viable options. It’s simple to implement and logically sounds. The requirement is simple, you need to know when the sheep in your server jump, but the server is not capable of getting to you and every one of you. So we set up a simple Javascript to auto refresh the page every few seconds.

The drawback is, page refresh is rarely favourable in the fast moving content packed web applications. By refreshing the page, you will need to refresh other resources needed where we are only interested in a simple integer that tell us if the Sheep has jumped. So, let’s move on.

Ajax Long Pooling

Ajax open up many great possibilities that until this day has become one of the core protocol in building responsive web applications. With Ajax, we can send a request without refreshing the page. This also enable us to implement a much more efficient auto refresh that doesn’t request the full page request. Instead, we send a smaller HTTP request in the background and update the page using Javascript.

In addition to that, we can set the connection to stay alive until we get a response (or time out). Anyhow, this will still involve the web browser to constantly sending new requests every now and then to get updated. One drawback is that the server might be idle for few hours without any new worthy updates, so the resources used to send those requests might just end up in waste.

Web Sockets / Comet / Probably other terms

lastly we have web sockets. With the web servers and browsers needing more and more frequents interaction, there is finally a way that the server can be event driven instead of request driven only.

Web Sockets allow a more interaction connection where the server can be allowed to send response based on events that happens in the server itself. In this way, the web browser (client) doesn’t have to constantly send request for updates, but simply be event driven as well that only reacts when needed.

This will also reduce the overhead of constant new requests that happen between the client and server.

However, web sockets may cause some compatibility issues depending on the server technologies used and also the browser. But we’ve seen an increasing support for this technologies that allow developers to create more and more responsive applications.

One popular technology that works well with Web Sockets is Node.JS. But there are ways to work around Django through Redis or other library that can provide us with this capability.

Reduce SQL Injection Risk in Python and psycopg2

It will be surprising that a slight different in your line of code can have a great impact in preventing SQL injection. Try to spot the difference between these 2 lines below.

# code 1 (with python string format() function)
db.execute("select * from some_table where filter = '{}'".format("going to Sam's party"))
# code 2 (with psycopg2 sql prepare syntax)
db.execute("select * from some_table where filter = %s", ("going to Sam's party"))

It probably looks obviously different, but for a second it looks like it shouldn’t give much different result. But sadly it does.

The first code use the python standard string formatting feature where given a string containing curly brackets as a placeholders like “This is a {}”, and with the format() method, it will fill those placeholders with other strings.

# Example
sample1 = "this is a {}, and also a {}".format("pen", "weapon")
print(sample1)
 
#results
# this is a pen, and also a weapon 

this looks fine for now. But try to do one for the string that we pass into the db.execute() above. If we print the string, it will give you the result below

select * from some_table where filter = 'going to Sam's party'

Notice the extra single quote on the filter? This will cause error and of course opening a whole world of opportunity for SQL injection attack. With the single quote unescaped, the rest of the string can be injected with other commands that will cause serious maintenance headache.

-- example: imagine the replacer string is "bleh'; drop table some_table; insert into user values ('some new malicious users'); --"
-- your query will become
select * from some_table where filter = 'bleh'; drop table some_table; insert into user values ('some new malicious users'); --'
-- note that double dash (--) is used for commenting. So the extra single quote will be ignored.

So, why does code 2 is a better way of string replacements? Because it has built in special character escaping mechanism in which all strings that are passed thorough this method will remain as a string instead of becoming a malicious codes.

db.execute("select * from some_table where filter = %s", ("bleh'; drop table some_table; insert into user values ('some new malicious users'); --"))
the code above will produce sql below

 

select * from some_table where filter = E'bleh\'; drop table some_table; insert into user values (\'some new malicious users\'); --'

Slow fgets in PHP. or does it?

We have a piece of codes that looks something like this

...
$old_file = fopen($filename, 'r');
$new_file = fopen($new_filename, 'w');

while($buffer = fgets($old_file)) {
  ... \\ $buffer is being edited here
  $buffer = $new_buffer;
  fwrite($new_file, $buffer);
}

...

It all works fine and very speedy all the while until one day we decided to run this piece of codes from a network drive.

Oh ya, did I mention that it’s running on a Windows Server?

Good.

When the codes above is running on the Network Drive, it slows down to almost 1 minutes per file, which normally only takes 1 seconds or less on a local drive. What went wrong?

Probably just like any debuggers will do, I went and print the execution time needed for each blocks of codes. Given that a while loop is a while loop, I didn’t put a timer inside the while loop, but only before and after the while loop.

The conclusion was that this particular while loop is taking most of the processing time. And as what most modern programmers will do, we ask Google about it.

“PHP fgets slow on network” I searched. Lot’s of results are complaining that fgets is slow. Hmm, now I know. But what’s the alternative? many. But hmm TL:DR. Too long, let’s find another shorter more to the point article (probably not this one I’m writing).

Searching on I read somewhere suggesting to comment out the fwrite() and try again. I did that, and it works faster. And of course it doesn’t solve my problem, but it seems that I’m going the right way. And voila, move the fwrite outside of the while loop, and it goes back to the better speed.

...
$old_file = fopen($filename, 'r');
$new_file = fopen($new_filename, 'w');
$buffer_holder = "";
while($buffer = fgets($old_file)) {
  ... \\ $buffer is being edited here
  $buffer_holder .= $new_buffer; 
}
fwrite($new_file, $buffer_holder);
...

So in short, don’t write into file line by line, write them at once. This way, you don’t have to go back and fort from Malaysia to US, but one trip is enough.

a real man only takes one trip

From Unicorn to Unicode

What is worse than knowing that Unicorn exists in some other dimensions but you will never be able to see it?

My answer will be the xA0 character from some encoding world that I don’t even know to exist. Being an Earthling, the only encoding world I’ve been and known is the Unicode. More specifically the UTF-8 realm.

Interestingly, many Unicode based systems reject the xA0 (or any nonconvertible characters) and totally crashes the system. Take Python for example, and also PostgreSQL later on.

Python

In Python, there is a function call unicode() that convert a string from other encoding to Unicode.

unicode(object[encoding[errors]])

However, the “errors” handling is defaulted to “strict”. It means that it will complain that “Something is wrong” whenever there is an error. Basically it means that it will break the system when there is an untranslatable character in the object that you are trying to convert.

There are two other options in handling conversion errors.

  • “replace” to replace the untranslatable character to the official Unicode replacement character
  • “ignore” basically replace the untranslatable character with an empty string.

PostgreSQL

When inserting non Unicode strings into an UTF-8 (Unicode based) databases, PostgreSQL will try to translate them first. Same thing will happen if the said string contain an untranslatable character, it will throw you an error.

This can be a hell of a problem because it technically break your system if your system is a one of those systems that process input and save them into a database.

So the solution is usually to try to catch these unicorns before they escaped into the database.

The adventure of the Old Mac line breaker in the Python world

There are many representations of a new line, End Of Line indicator, or a line breaker. You probably heard of the terms Line Feed (LF) and Carriage Return (CR). They are technically characters like the letter “A” and small letter “a”. But instead of printing the letter, they tell the system that it’s the end of a line. However, different computer system uses these 2 common characters in different ways but let’s narrow it down into the 2 most common ones, namely the Unix version “LF” and the Windows version “CR+LF”. But wait a minute, there is this Old Mac version as well that uses only CR character to represent the end of line.

Interestingly in the Python’s universe (and probably some other even more racist universes), the Old Mac convention is by default not a line breaker. If you read a file full of lines that only ends with “CR” using the standard file open() function in Python, they will come out as a single line text.

As a slightly less racist developer, we need to build applications that can support as many types of stuff as possible. Here are 2 tricks to help you ensure the file you are reading is read properly the next time you use it.

When reading a file

# Use the 'rU' mode so it understand the Old Mac properly
file = open('filename', 'rU')

If you happen to be working with File upload in Django, this might be useful

# http://stackoverflow.com/questions/1875956/how-can-i-access-an-uploaded-file-in-universal-newline-mode

# First, read the uploaded file and convert it to unicode using unicode() function
# Second, stream the file using io.StringIO function with the Universal-newline mode turn on by setting newline=None
import io
stream = io.StringIO(unicode(request.FILES['foo'].read()), newline=None)

Slicker way to export crosstab in Tableau

Tableau has an awesome feature to allow users to export the processed data into crosstab, or I would say Comma Separated Value format.

We use Tableau to help us visualize data into graphs and charts. And our raw data is often messy and huge and doesn’t really make sense in that granular state. So the charts and graphs are really much better to look at. And even though some users are happy enough by looking at charts, some others find it important that we can get the post-calculated raw date, i.e. the numeric equivalent of the given chart.

Image

But when it comes to a grander scale of a few thousands rows of records, The Export Crosstab to Excel just doesn’t work well. It took a very long time to work, and that if it works at all.

So, instead of exporting it to Excel, use Copy -> Crosstab function. This function is surprisingly much faster than exporting to Excel.

Image

As a comparison, exporting 50,000 rows to Excel will take forever, and never but copying the same 50,000 rows to clipboard only takes 5 seconds.