Phelio Gnomi

My words from my mind

How I forced myself to spend time on a side project

I found it so much easier for me to be productive when I’m at office, forced to work from 9-5 every Monday to Friday than to working at home on a side project where I don’t really have a fixed working hour.

Why is it so hard?

Few differences that I observe to be different between office and home and how I fix them:

You are mostly working alone at home.

Most of the side projects I’ve done is a lone project. Somebody (or me) started it and a developer is needed and I volunteer and boom, you are all alone. It could be a good thing for some people, but I find it very demotivating.

Well, most projects failed badly and I blame the loneliness that I felt during the project. I felt that I needed approval from another peer. I needed somebody else to say that, “Hey, it’s awesome”. Or, “hey, you can do this better by doing this”. You have peers probably scrutinizing your work all the time at office. But when you don’t get the scrutiny, you don’t feel confident.

But hey, again, you are the master of everything now. It’s your own project and you know best about everything. I need to constantly remind myself that “I’m a good programmer” and “it’s OK to make mistakes”. It’s easier to say than done, of course. But, it’s really normal to fail. Failure is how we learn to be better.

Another thing that might help is the online community. Go to forums, stackoverflow, or github. Share your works, and get feedback or get some peers to help you do the things that you are working on. It’s kind of popular right now, isn’t it?

You don’t have a fixed time at home.

When you are at home, you start working when you feel like working. But there is always a problem to start working. I found that the problem with me is that I am lazy. And the reason of why I’m lazy is because the amount of time that I need to spend working is infinity. Unlike at office, you have a finite working hour everyday. You know that you just need to work until 5 o’clock and after that you are free. And that kind of make you happy a little bit.

Do the same thing at home. Set the amount of time you are willing to spend on your side project. For me, the easiest way to do so is by setting a timer. In fact, I have a timer running right now as I’m writing this. Set it to one hour each, don’t make it too long. Just enough for you to finish a small task.

So many distractions at home.

You got so much freedom at home that you probably won’t start doing anything. That’s certainly not true. I’m supposed to be working on my side project right now, but instead, I’m writing this article. Let it be. It’s something productive anyway. Don’t feel bad about yourself. Do things that you enjoy doing. If the project never got finished, at least you’ve got some fun time doing something else. Don’t to be too harsh on your life. There is time for playing, and there is time for working. When the time for working comes, you will find yourself working.

You don’t directly get paid when working on a side project.

Well, this is probably the hardest part. Money, either directly or indirectly, is probably the greatest motivation of all. It’s really hard to do something unless you are certain that it will give you something back in return. And that something is normally money. But, for personal project, you are normally getting the money at the very later stage of the project. Probably after you’ve spend thousands of hours and bloods (and money).

That’s why you need to swift your motivation a little bit. Start by thinking that money is not what you are aiming for. Let the project be a “just for fun” project without shutting its prospect to shine and become a gold digger. In other words, only do project that you enjoy doing unless you get paid upfront.

The art of writing codes

I’m supposed to be fixing a bug in my Expressionengine module right now. But, instead of that, I feel like procrastinating a little bit and write about fixing bugs. Not exactly just about fixing bugs, but a lot of other things about writing codes. But well, I think I didn’t mean to write about bugs, but since it’s been written this way, I will just leave it there. I will write about fixing bugs next time.

This time, I would like to write about writing. Writing is a very old technology. Writing is said to be the requirement of a civilization to success. And now, we have come to a point of time where writing is very important for every single person on Earth (I’m not really sure about other planets, though).

Holy crap, this is going to be a very boring article.

There are probably tons of articles on how to write English well. My question is, can we apply these human language principles in programming languages? why not?

Writing codes for computers has evolved from writing symbols that only computers can understand (or probably binary? I don’t know) to the variety of so many high level programming languages where it’s more for human to read rather than the machines. But the problem is, there is still a very big gap between what a computer can understand and what a human can read. And I have to say that it’s easier to write for the machine than for human to read.

My job for the past few weeks are basically reading codes. I would like to call my self a software archaeologist.  Wait a moment, some other dude already used that term before? fine…

Reading people’s code is time consuming, stressful, difficult, and a very complicated job. Programmers love to complain about having to debug some buggy codes or  being in the maintenance mode without being able to write new codes from scratch. But, let’s read it like a book. There are stories to unfold, mystery to solve. There are characters like functions, objects, variables, classes. the more we read, the more we understand them. The emotional connection between the readers and the fictional characters grown.

What if we write codes like we write books. What I mean here is that we take human as our main audience and focus on delivering information that are intended for human to read. That’s probably what we have been doing all this time by writing comments in our codes. But, writing comments are just part of the task here. I believe we can do much more than writing comments. We can even write to a point where we don’t really need to write any comment anymore.

Proper naming of your characters

A good writer doesn’t roll a thousand faced dice filled with human names and set their characters based on the names that show up. Most writer pick a name because it has something to do with the story. Darth Vader actually means Dark Father where he turn out to be (Spoiler alert!!) Luke’s fathers.

We name our everything in our codes. We are the god of that universe. But that universe is not yours alone. So, name them for the purpose of clarity. We want the readers to understand what a function does without having to rip open that function. In other word, generate_random_name() should generate a random name and kill_your_self() will actually kill your self.

I know it’s difficult sometimes to pick a proper name. But, at least think about the future human readers when you are naming them.

Human readable algorithm

Once you’ve got the naming right, it’s easier to write human readable algorithms.

if andy.isStupid:
  andy.go_to_school()

But, there are also some other things to consider. There are always more than one way to achieve a goal in coding. The kind of codes I would like to read are the ones that are easiest to read without compromising performance. At the end, speed is still the most important thing for your boss.

Spaces

Put a lot of them. Put a line break on every groups of codes. Make it clear that this chunk of codes are family while its neighbor down there is something else. Remember, you are writing for human to read, not the machine.

Consistency

Last, and the most important one IMHO, is make it consistent. Make every style in your codes mean something. If you like camelcase, use camelcase. I sometimes use camelsase and under_score for a different purpose. camelCase for variables and under_score for functions.

It’s never about right or wrong. It’s about trust. If camelCase means functions, and suddenly I see camelCase which is not a function, I lose that trust on you. I can’t trust you anymore and it makes the whole program harder to read.

From rwx to 777

Have you ever come to a situation where you are so new in town and you don’t know how to blend in? You kind of see everybody as a codes that’s made up of rwx rwx rwx but you are speaking in the language of numbers like 667 or 557.

file premission

Now that you need to be like everyone else, I’ll teach you how to extract the numbers from those “rwx”es. First, you need to catch one of them, bring them into a dark room called stat. Once he’s inside the stat room, he will speak out many other languages, including the number language that you need to know.

stat something.a

code

 

Now you know their number. You know what to do from there.

If you still don’t know, here is the magic words.

chmod <the magic number> <yourname>

For example:

chmod 0776 you

Speeding up your query with Temp Tables

If you have a query that:

  1. have a lot of subqueries
  2. join many different tables or subqueries
  3. going to return a huge amount of data
  4. take forever to run (more than 1 minutes)
  5. you have a busy database server

You can run your query in a hundred times faster speed by storing some groups of subqueries results into a temp table first before running the final joins using these temp tables.

This way, you will have:

  1. a smaller sets of queries to run
  2. which will reduce the memory usage
  3. and reduce the chance of blocking other running queries
  4. and it runs a hell lot faster.

Let’s make a super simple example.

--Old and Slow
--let's assume that huge_table_a and huge_table_b both contain 1 billion records each
SELECT tbl_a.item,
       tbl_a.profit_share,
       tbl_b.profit_share,
FROM   (SELECT item,
               Sum(profit) / Max(profit) AS profit_share
        FROM   (SELECT item,
                       profit,
                       UNNEST(regions) AS region
                FROM   huge_table_a
                WHERE  region = "south")
        GROUP  BY item) tbl_a
       JOIN (SELECT item,
                    Sum(profit) / Max(profit) AS profit_share
             FROM   (SELECT item,
                            profit,
                            UNNEST(regions) AS region
                     FROM   huge_table_a
                     WHERE  region = "north")
             GROUP  BY item) tbl_b
         ON tbl_a.item = tbl_b.item
         ORDER BY tbl_a.profit_share DESC;

Using Temp Tables

-- Will run a lot faster especially in a busy DB
CREATE TEMP TABLE tmp_table1 AS (
       SELECT item,
               Sum(profit) / Max(profit) AS profit_share
        FROM   (SELECT item,
                       profit,
                       UNNEST(regions) AS region
                FROM   huge_table_a
                WHERE  region = "south")
        GROUP  BY item
);

CREATE TEMP TABLE tmp_table2 AS (
       SELECT item,
               Sum(profit) / Max(profit) AS profit_share
        FROM   (SELECT item,
                       profit,
                       UNNEST(regions) AS region
                FROM   huge_table_a
                WHERE  region = "south")
        GROUP  BY item
);

SELECT tbl_a.item,
       tbl_a.profit_share,
       tbl_b.profit_share,
FROM   tmp_table1 tbl_a
       JOIN tmp_table2 tbl_b
         ON tbl_a.item = tbl_b.item
         ORDER BY tbl_a.profit_share DESC;

 

Better way to execute sql files in PostgreSQL

My previous post  is kind of dumb. It’s simple, but dumb. First of all, the /o command will only write the output of a particular query into file. It won’t write any error or any other information that pop out during the execution. The last time I ran that, it failed and I don’t know what happened or where it stop.

So, a better way to do it is by running psql with -f option.

psql -h HOSTNAME -U USERNAME -d DBNAME -a -f FILENAME > LOGFILENAME 2>&1

A little explanation here:

  • -a tells the psql to print out all the input commands to Standard Output. It basically print out all the queries inside your SQL file. This is handy so that you know what went wrong and where did it stop.
  • -f FILENAME tells the psql to run the given file
  • > LOGFILENAM is the basic UNIX command to redirect the standard output to a file. In other words, it says that whatever output that you are supposed to see in psql will be written into a file instead.
  • 2>&1 is very important because it asks the system to also print any errors into the standard output. Since we’ve redirected that standard output into a file, it means that any error will be logged into that file also.

For more information, checkout the psql manual and also the Bash redirection.

PostgreSQL Input and Output

NOTE: this is one way of doing it. Find the better solution here.

Let’s imagine a scenario where you have 10 different SQL queries to run and they will take few hours each. You don’t want to run it in the peak hours because it will slow down your entire server. You don’t want to stay back late either. So you decided to put all the queries in a text file and run them with your PostgreSQL client at the end of the day and you go for a bottle of beer at the nearest club and call it a day.

Here is how you do it in PostgreSQL. First, fire up your PostgreSQL client such as

psql -h your-host-name -U your-username -d your-dbname

and then, you run the \i command. Well, “i” is obviously stands for “input”.

\i /path/to/your/sql/file

VOILA, the sql are running. And then you pack your stuffs and call up your friends to meet you up at that bar you talked about earlier. But wait a minute. What if something went wrong? what if my console suddenly close itself? I won’t know if all the queries ran successfully or not. Don’t worry, you say. We can stream the output into a log file. And when we come back tomorrow morning, we will know where to check. And so, you pressed CTRL+C to cancel your previous job and waited a while until it finish cancelling it and then you start stomping some other commands:

\o /path/to/your/log/file
\i /path/to/your/sql/file

And voila!! you can really go to that pub where your friends are waiting now. Have a good evening, folks!!

The theory of Object Oriented Programming vs its application.

Speaking of Object Orientated Programming (OOP), I remember how absurd it was to learn for the first time. In the university, they first teach you a bunch of concepts, ideas and terminologies. After that, you are on your own.

Being a totally newbie in the programming at the time, and a very horrible English speaker in a All English Speaking programming class, I was so proud to be able to use For loops in my C++ class. The same things goes to how proud I was to understand theoretically all the terms that the lecturer explained. I was so proud to get it right about the hierarchy of the daily product family. And the similarity between how your father will left you some fortunes when he died and when a child class inherit a mother class. Theoretically, I was an expert.

When it’s time to apply the theory into practice, everybody was left amazed by why you don’t create a Milk super class and have cheese, butter, and yoghurt extends the Mother Milk just like the Hierarchy diagram explained. You don’t create any classes for them, but instead, you created a class called <<ProgramName>> and store the daily product in the database.

The OOP era ended for me pretty quickly because I was introduced PHP and suddenly I’m happy with my life without thinking much of making the codes look more like what it does in the real world. Carrying the OOP theory in the very deep side of my backpack, I fast forward 5 years, surviving finely. Throughout my working history, I’ve never been in a company that really embrace OOP.

However, things is going to change. The first time ever in the real world I was in a job interview where the people actually care about how much I know about OOP. Digging back into my memory, I managed to answer some of the questions correctly and miss some. I have to be honest with them that I have no experience with OOP at all. But they offered me a job and now I’m happy that I can finally create the Milk class that I didn’t create 5 years ago.

For the moment, OOP is still a very vague concept for me. But I’m looking forward to apply those theories into real practice this time.

Back to Yii

I finally solved the problem with the Selenium testing. The problem wasn’t because of the “Failed opening ‘SiteTest: Firefox.php’” at all. It was just a warning message. The real problem is the curl extension in PHP. I thought I already enabled it, but not when the CLI version is not using the same ini file. So, after I enable the curl in the CLI version of the ini file, it works fine.

I’m still wondering though, how do I make the CLI version use the same ini as the web version.

I wish that I’m done with Yii

No offence Yii, but I’m going to move on. First, your lack of examples in your user manual – where the the manual itself is mediocre to start with – is tolerable. But, I started to wonder if people really use you. For a tool like you, you should be proud if people talk a lot about you, especially best if they talks about the problems they are having with you. At least by then, there will be lots of devoted worshipper that are willing to find the solutions for every problem that people talk about.

I just can’t do it any more. I’m stuck here with a problem that nobody (that I can reach) knows. It all started with one very promising book “Agile Web Application Sevelopment with Yii 1.1 and PHP5″ which is really good to start with. But it only take 3 chapters for you to left me bleeding at a corn field near my uncle Jimmy’s house. I’m still bleeding by the way, just so you know. I just can’t figure out how to make the Selenium works without giving me that “Failed opening ‘SiteTest: Firefox.php’”error. I can’t proceed without Selenium, the whole book is about Test-driven approach. And if the testing tools doesn’t work, how to proceed?

There are still plenty of fishes in the sea. Though you are the chosen one and I have to come back to you one day, I think I’ll just forget about you until then.

The Ruby, C++, PHP and C++

I’m taking the Stanford Online Course called the Design and Analysis of Algorithm 1. So far so good, and then the Programming Exercise 4 hit me hard.

First of all, we are allowed to use any programming language. They only want the final result, regardless of how you process them. So, at week 4, we are supposed to read through a file that represent a Graph with thousands of nodes. And we are asked to find the five largest strongly connected components or SCC.

So, I started with Ruby, which is my language of choice for this course. The algorithm that the professor teaches use recursive. So, after finishing the program and running it trough few test cases which seems to work properly, I run it on the actual file (the thousands of nodes graph). It crashed in the first few seconds saying that “stack level Too deep”. Apparently the recursive is too deep for my Ruby to process. I tried to find how to increase the stack level allowed, but it’s either I don’t understand what they are talking about or the solution is for Mac or Linux. I’m working on a Windows 7.

So, I think maybe I should do it on C++. One reason is because it’s fast, and it’s difficult (so I can learn to be a better programmer, not just a guy who get pampered by modern programming language). Then, if you know C++ and also know Ruby or PHP, you can see that C++ is so much different. So much more difficult. PHP pretty much take care of many things behind the screen for you,things like dynamic array or array with random index name (hash in Ruby, associative array in PHP), or passing the array around like tossing pancake from the pan to the plate. They are not that simple in C++. Array is just a pointer with the size. Passing array is complicated, not to mention about multilevel arrays. It doesn’t have hash or associative array.

I have to admit that I’m too pampered by modern (high level) programming language and it’s not the C++ fault, and creating your own function to search an array or to enable dynamic index naming is just too much for me. So, I go back to PHP and try out it’s maximum stack level allowed. First try, it return error on the 100th recursive. But don’t worry, it turned out that it’s just the limitation that X-Debug set. I remove X-Debug and it can run for as deep as your computer memory can supply (which is a lot and a good news).

I immediately converted the Ruby code into PHP code and voilà, it works on the test cases. Working on PHP is like returning home. It feels natural for me as if I’m talking in my mother tongue. However, When I run it on the real graph, I hit a lot of memory limit error. I increase it few times, from 128M to 256M, to 1024M to 2048M. It consumed more than 1GB or RAM to process 5 millions connections of 800 thousand of nodes. And it run forever. And ever. I never get the result back yet. And I’m suspecting an infinite loop, but can’t really find the proof of it. Therefore, I have to conclude that it’s the PHP problem. It’s simply to slow.

So, my only hope is C++ now, which I’m not good at, and I have to build many functions from scratch. Well, probably I can find some library online. But, I don’t know. It feels like I’m forced to speak Spanish when I only know a few words like “la nina come pan” and “el niño bebemos leche”. And I’m literally not sure is it bebemos or beben? Oh, I’m screwed.

Follow

Get every new post delivered to your Inbox.