That Guy From Delhi

18 Dec 2015

Getting my hands on a Google OnHub

So finally, I get to lay my hands on the new Google OnHub at home.

Unlike its other Google cousins, the OnHub isn't yet available for sale in India and then its a niche product (yet) here in India. The other obvious question is whether you'd pay 6x the price for a router, even when a brand like Google sports it.

I think I would and so was finally able to put my foot down when I realised that I had had enough of other routers making things difficult at home.

A few features I liked:

Automatic Software update

I loved this aspect that is pretty much missing in all its contemporaries

Considering that my 10 year old D-Link 502T hasn't received a single firmware / OS update I was scared to death what all crapware was running on my Home WiFi a month back.
That coupled with a few Einsteinish Router companies forced to admit secret (idiotically planned) backdoors, it just isn't funny to realise that my Home Router was probably a 'piece of cake' for a script kiddie trying to login.

Prioritize a phone

Again its a pleasure showing it to my wife how easy it is for her to prioritize her phone, when the kids are watching YouTube in HQ.

Router configuration a breeze

Its super simple to manage

I just recently got a Chromebook for Audio replacement working at home, and it was pleasant to realise that setting static IP address wasn't about setting /etc/network/interfaces anymore. The PI2 stayed on DHCP and I set the OnHub to give the MusicBox a static IP hereon... QED :) !

Manage your OnHub from China!

Once configured, you could manage your router sitting hundreds of miles away!

Which basically means no long calls to your GrandMa asking her to read out what is on the screen when she types 'http://192.168.1.1' on the browser.
You could be managing multiple OnHubs on your phone, each sitting at your parents place hundreds of miles away (without VNC / TeamViewer / RDP hacks) and still configure every minute detail such as setting Port-Forwarding / DNS / DHCP etc. from your SmartPhone.

WiFi connection optimization

Frankly, with so many walls, some remote corners of my house have seen some network quality degradation at times, but I haven't seen a 'No Network' message yet. So probably its doing a good job there, but I am sure I can't tell that right away.

Add to that, if we consider that this machine is a dual core machine (with a GPU) most of which isn't even put to use (yet), I am pretty excited to know what its real potential is and how Google upgrades my 'boring router' down the line.

Rumour is that this might just be a Google's shot at an Echo or a Siri sitting in your drawing room. But till that happens, I'd have to stay pleased with a beautiful router sitting on the desk :)

Now you may want to get paranoid and all and worry about how Google could keep an eye on Dr. Lanning (you), but I have a feeling that it'd take a while before I give breadcrumbs to a Detective Spooner.

All in all, a (pretty) costly router upgrade but I ain't regretting it.

17 Dec 2015

MusicBox + Pi1B + Ancient 4.1 Speakers => Chromecast for Audio

Since Chromecast-for-Audio hasn't yet been launched in India, eBay sellers are using near monopoly extra for such a device here. Frustrated with being blackmailed like this, I resorted to conjuring up a combo that finally gave some shape to my 2 year old Pi and a ten year old 4.1 music system a life, and gave a good groove to my (otherwise boring) drawing room :) !
to charge a painful 150%

Parts:

Any Raspberry Pi

Except PiZero

Too low powered, would not work

To clarify, either of the following combos would work

The old PiA+ / PiB+

with SD Card

2GB or more

The newer Pi2

with MicroSD Card

2GB or more

MusicBox
Win32DiskImager
Client Operating Sytem

(Desktop) Windows

Bonjour

Optional

Makes the URL easier

I had Apple iTunes that install Bonjour support by default

(Smartphone) Android

Doesn't support Bonjour out of the box
Alternative,

Get the Pi to come with constant IP address

Very easy with my new OnHub

Bookmark the URL instead

Volume Control

The Web Interface allows changing volume
From the command line you could try this

amixer cset numid=1 -- 80%

Happy Listening :) !

20 Nov 2015

Reverse Port Forwarding and how !

Recently had to work on a PoC that required spinning off Docker instances with custom configurations. The problem wasn't as much with the Docker aspect, which is probably up for another story sometime, as much as the fact that as luck had it, we couldn't commission a separate machine for this PoC, and I eventually had to do the entire development inside a 2G VirtualBox VM on my 8G Laptop.

The tricky part was that we had to submit the URL to the PoC Landing Page even before it could be ported to a full-fledged VM/server.

So effectively, we:

Submitted a URL (on ServerA) running Apache, but no space to host a 40G VM.
Had a Laptop where the VM was developed, but can't stay online since its not always connected
Had another server (ServerB) where the VM needs to be ported, to be always up for PoC evaluation
ServerA was in the US, whereas, ServerB (& Laptop) are half-way across the world in India.

Separately, the VM was an Ubuntu installation that had the following:

A running NodeJS (Bootstrap / Express combo) serving the PoC GUI
Separately it hosted the docker runtime that spun off Docker instances that had their own propietary application + Tomcat running on port 8080, all of which eventually supposed to be visible company-wide
Thirdly, it had a PostgreSQL 9.3 database that served as a backend to all the running docker-based app instances

To connect ServerA with the PoC hosted on ServerB we tried the following:

Creating VirtualHost entries on the ServerA that pointed to corresponding ports on the VM

This failed because the VM was running inside a NATted configuration and thus was invisible to ServerA

Then we tried an alternate solution, wherein Apache->VirtualHost entries on ServerA were pointed to port 80 on ServerB. This required port-forwarding port 80 on ServerB to port 3000 (NodeJS) on ServerB. Although for this we used NetSH, this failed for various reasons, some of which we could identify and some we couldn't. For e.g. we installed the ipv4 module for NetSH, also tried installing ipv6, as well as, enabling Windows Firewall for NetSH to work (contrary to popular belief that Windows Firewall might be the reason for things getting blocked)
We finally settled down to running persistent SSH connections directly from the VM to ServerA (thereby by-passing all Windows Firewall issues) and setting up Remote Port-Forward such that remote ports could connect to Node Server inside the VM

This too initially didn't work as expected. To identify the cause we:

isabled SELinux on ServerB, then ServerA (although that didn't make sense) as well as on the VM to no avail
Then we disabled IPTables on all the relevant machines and that still didn't help
Finally we realised that the Remote Port-Forwarding though working well, was getting attached to the localhost interface on ServerA. Which means that although 127.0.0.1:80 on ServerA was working as expected, 192.168.1.1:80 on the same server immediately gave a "connection failed". It took a while to realise that though I was logging in trans-atlantic, the immediate 'Connection Failed' was an issue with ServerA and not with this side of the line.
What complicated things post that was that the regular -g option was insufficient. It required setting the GatewayPorts on the SSHD (with a sshd restart) to work as expected.

Eventually, after a few hours of transatlantic jugglery, finally got a working URL that looked something like this... What a day !

12 Nov 2015

Pigz - Parallel GZip

For a while, I've been frustrated with the fact that GZip was unable to consume idle CPUs on my laptop. Recently read about PIGZ (a drop-in GPL replacement) that implicitly consumes all CPUs.

Pronounced Pig-Zee, PIGZ is a drop-in replacement for GZIP and automatically consumes all CPUs of the host machine in order to get the compression completed much faster.

OLD: tar -cf - sourcefolder | gzip > sourcefolder.tar.gz

NEW: tar -cf - sourcefolder | pigz > sourcefolder.tar.gz

Enjoy!

11 Jun 2015

Using Pi as an home-based Media Server

This is among a series of articles on my experience with the Pi.

This article is about using a Pi as a primitive low-end File-Server for your home network:

Expectations:

Torrent-Server

Download Torrents
Store on a File-Share

Windows Share

Serve the File-Share as a Windows Share Drive
Allow Read / Write to this Windows Share Drive

Use File-Share as Media-Server

Using any Smartphone / Laptop

Play Movies using VLC
View all Photos / Home-Videos

Effectively:

Always on

Always accepting new Torrent requests
Instantly start downloading

In Real-time

Allow user's to view torrent download status

Use any UPnP Phone App to play Video content over WiFi on a SmartPhone
Use VLC (Network Streaming) to play any Video over WiFi on Laptop / Desktop
Use Windows Share Drive and view all Photos / Home-Videos as needed

How-To:

On Server

Install Torrent-Daemon
Configure Torrent-Daemon to listen on RPC requests

Here's the howto for that

Configure Samba

Set the Download folder to be shared

On Windows

Install Transmission-GUI-Remote

Desktop / Smartphone

Configure GUI to use the RPC based Torrent-Daemon server
Make this application the default for .torrent & magnet files

On Linux

Install Transmission-Remote

sudo apt-get install transmission-remote

Configure that to use Daemon (instead of downloading directly with transmission-cli)

Pros/Cons:

Pros

Once torrent download has begun, client can disconnect / shutdown client computer
Server continues to get torrents, after a restart
miniDLNA serving speed pretty decent

Watching a movie (over WiFi) on VLC

CPU ratio barely 0.01 which is pretty decent

Should be able to easily serve a small army :)

If no other IO is happening
If WiFi isn't the bottleneck

Cons

Storage on Pi needs to be managed from time-to-time

Currently is highly adhoc based

Truly taking RAID concept to heart!!

Have 6 Pen Drives (ranging from 4GB to 16GB)

8GBs are INR ~170 ( $3 )
All connected via 3 USB Hubs (both USB 2.0 and USB 3.0)

One Powered, two non-powered hubs

Most on Btrfs

Pretty stable, surprising that still get 1+ MBps with Pi's CPU

Some VFAT fs since have need of moving stuff off of Pi to a Windows Laptop
All mounted under /disk

Very temporary arrangement

Ideally am looking for a LVM solution (which allows me to remove / add pen-drives on the fly and thus can be called as one) that actually is suited for this purpose.

But miniDLNA picks up Photo / Audio / Video pretty well from all different mounted folders

Download speed limitations

Internet Speed = 2 MBps
Daemon Download speed capping = 1Mbps

But PI never reaches it :D !

Just imagine the poor configuration

Pi + Btrfs + USB 2.0 + 3 USB Chain + Unpowered Hub

Still consistent at 450kbps! Impressive!

Lack of Transmission-Daemon configuration tool means all configuration has to be done via black-screen configuration files

Careful configuration

Give all access to 'all' users only if you're sure that all users are going to be careful

25 May 2015

SAVE Tax with new NPS provisions!!!

This article is about NPS (National Pension Scheme) or New Pensions System as some may know, which is a loose-ended replacement of the erstwhile Pension System prevelant in India. Although not a 100% replacement, this is the next best thing, and thus understanding it better is a good idea.

For all those trying hard to save Taxes while working their way around the Indian Income Tax system, the following may help in clarifying doubts / providing more avenues.

As per Income Tax provisions effective in 2015-16, investments in NPS can be done in three ways:

Employee contribution, under 80C
Direct investment in NPS (outside of your employer), under 80CCD(1B)
Employer contribution, under 80CCD(2)

Let's look at these in more detail:

Employee contribution, under 80C

Any Indian (that has a PRAN) can invest in NPS by contributing to their NPS account by investing a minimum of Rs. 500/- per month or a minimum of Rs. 6000/- annually. This investment into NPS can be deducted from the person's taxable income (subject to a maximum capping of Rs. 1,50,000/- capping under 80C).

Although a good idea, this however, is generally futile since most people (who have been working for a while), have already reached their Rs. 1,50,000/- 80C limit via other means (for e.g. Life Insurance / ULIP / PF etc.). Then investing in NPS is although good in the long-term, however, it does not contribute to saving tax for the current financial year.

Direct investment in NPS (outside of your employer), under 80CCD(1B)

Any Indian (that has a PRAN) can invest directly into NPS, without the support of his / her employer. This facility has been available for a while, and is the oldest form of investing in NPS. Till recently, the caveat to this form of investment was that this did not have any Tax Exemption.

Although modified earlier, as of 2015-16, the Income Tax Provisions are such that investments in NPS (made directly) up to Rs. 50,000/- can be deducted from the person's Taxable Income. To clarify, this doesn't mean that one can't invest more than Rs. 50k, but that only the first Rs. 50k of that amount can be deducted from his / her taxable income.

For e.g. Lets assume that Ms. Lata's has consumed Rs. 1,50,000/- 80C investment options (via Life Insurance / PF investments) her net taxable income is Rs. 3,75,000/-. Now lets assume that she invested Rs. 1,50,000/- directly to NPS (outside of her employer's assistance), then the next taxable income for her would become Rs. 3,25,000/- (i.e. 3,75,000 - 50,000). So although the entire sum of Rs. 1,50,000 was invested into NPS, only the first Rs. 50,000 was deducted from taxable income.

Employer contribution, under 80CCD(2)

The third and the most unclear & interesting section is the 80CCD(2) that allows an employee to save much more tax than was possible earlier.

Under this section, (apart from the above two clauses), an employee can request his / her employer to deduct a given sum from the monthly salary, and invest in NPS. This contribution (upto a maximum of 10% of Basic Pay) can be additionally deducted from the employee's taxable income, which in some cases can be a big boon to the net tax outflow in the financial year.

Example for all above sections

Lets take an example that elaborates all the sections given above:

Lets assume that Ms. Lata's Basic pay is Rs. 11,00,000 (11 lakh) and she has invested Rs. 1,00,000 in Life Insurance and Rs. 40,000 in ELSS Funds, as well as Rs. 10,000 in NPS (under section 80C). Further, she directly invested (outside of her employer's assistance) invested Rs. 50,000 in her NPS account (under section 80CCD(1B) ). Lastly, she requested her employer to invest Rs. 10,000/- per month in her NPS account under Section 80CCD(2).

Then her net taxable income would be as follows:

Taxable income = 11,00,000
                   - Rs. 1,50,000 under 80C      - max (1.5 lakh)
                   - Rs.   50,000 under 80CCD(1B)- max (50k)
                   - Rs. 1,10,000 under 80CCD(2) - max (10% of Basic)
   = 11 lakh - 1.5 lakh - 0.5 lakh - 1.1 lakh
     = 7.9 lakh

This should clarify all doubts pertaining to investment in NPS for the financial year 2015-16.

17 May 2015

Basic OLAP Support in PostgreSQL

While reviewing an existing application, I thought it'd be worthwhile to review how good / bad PostgreSQL is in terms of OLAP. This (growing) post is going to be my (un)learning of how ready is PostgreSQL.

Row Numbering

Support: Yes.
Use: Row_Number() function numbers rows generated in a result-set.
Example:

SELECT
row_number() OVER (ORDER BY marks DESC) AS rn,
name
FROM x;
Review: Some databases have different variants that accomplish this (for e.g. Oracle has a pseudo column called ROWNUM), but PostgreSQL fully supports the SQL Compliant syntax.

Rank()

Support: Yes.
Use: Rank() and Dense_Rank() functions number the rank of the compared item.
Example:

SELECT
rank() OVER (ORDER BY marks DESC) AS rn,
dense_rank() OVER (ORDER BY marks DESC) AS drn,
name
FROM x;
Review: Its useful and fully supported.

Window Clause

Support

OVER (PARTITION BY): Yes
OVER (ORDER BY): Yes
OVER (RANGE): Yes

Use: Read more here.
Example:
Review: These are extremely helpful for people serious about data-extraction / reporting and fully supported.

NTile

Support: . Yes
Use: Ntile().
Example:

SELECT
ntile(4) OVER (ORDER BY marks DESC) AS quartile,
ntile(10) OVER (ORDER BY marks DESC) AS decile,
ntile(100) OVER (ORDER BY marks DESC) AS percentile,
name
FROM x;
Review: Versatile and fully supported.

Nested OLAP Aggregations

Support: No

But doable with alternative SQL? : Yes

Is that as Performant? : Mostly No

Description: Allow something like

SELECT
subject,
AVG(SUM(marks) GROUP BY class)
FROM marks
GROUP BY subject;
Alternative: This could be done with Sub-Selects like this:

SELECT
subject,
AVG(sum_marks) AS avg
FROM (
SELECT
subject
class,
SUM(marks) AS sum_marks
FROM marks
GROUP BY subject, class
) mrk
GROUP BY subject;
Review: In the two examples we are trying to calculate the Per-Subject-Average of (Total marks obtained in different classes). Although PostgreSQL doesn't support this form of nested-aggregates, it clearly is a neat form of doing things. The alternative, acceptably looks like a kludge, and it would be a nice to have feature.

GROUPING SETS

Support: Yes (in 9.5)

Alternative: This could be alternatively done with UNION ALL like this:
SELECT SubjectID,
NULL AS StudentID,
AVG(marks)FROM marksGROUP BY SubjectID
UNION ALL

SELECT NULL AS SubjectID,
StudentID,
AVG(marks)FROM marksGROUP BY StudentID;
Review: Popular databases (Oracle / MSSQL) support this well. PostgreSQL has had this has on the ToDo list from at least a decade ! Looking at the alternative, one can see that this is not just lengthy (and repetitive .. thus error-prone), but also non-performant (simply because it requires multiple-runs of the same data-set).
History:

Already in PostgreSQL TODO list
Discussions started (at least) way back in 2003.
Patch:

A 2008 patch that didn't make it.
A 2014 patch was heavily in discussion since and finally just got through to PostgreSQL 9.5.

ROLLUP

Description: An obvious extension to GROUPING BY (explained above), ROLLUP could be explained with a simple example:

GROUP BY ROLLUP (Year, SubjectID, StudentID)

is equivalent to

GROUP BY GROUPING SETS
(Year, SubjectID, StudentID)
(Year, SubjectID)(Year)
();
Support: Yes (in 9.5)
Alternative: This could be alternatively done with CTEs.

WITH x AS (
SELECT Year, SubjectID, StudentID, marks
FROM marks
WHERE passed
AND NOT inactive
)
SELECT *
FROM x

UNION ALL

SELECT
Year, SubjectID, StudentID, AVG(marks)
FROM x
GROUP BY Year, SubjectID, StudentID

UNION ALL

SELECT
Year, SubjectID, NULL AS StudentID, AVG(marks)
FROM x
GROUP BY Year, SubjectID

UNION ALL

SELECT
Year, NULL AS SubjectID, NULL AS StudentID,
AVG(marks)
FROM marks
GROUP BY Year;
Review: ROLLUPs are ideal to generate things like Sub-totals, which at times form key performance factors when generating large Reports. The alternative essentially uses a CTE, which is subsequently used to calculate subtotals and totals. For multiple-reasons, this is sub-optimal and can be sped up, if only for in-built support. Besides, the alternative is lengthy & repetitive (thus error-prone).
History:

Discussions started (at least) way back in 2003.
Patches submitted

The 2010 patch seemingly didn't make it.
The 2014 attempt finally got through.

CUBE

Support: Yes (in 9.5)
Description: Just like ROLLUP (was an extension of GROUPING SETS), CUBEs are an extension of ROLLUP (and thereby GROUPING SETS) and could be explained with the following example:

GROUP BY CUBE (Year, SubjectID, StudentID)
is equivalent to

GROUP BY GROUPING SETS
(Year, SubjectID, StudentID)
(Year, SubjectID)
(Year, StudentID)
(Year)
(SubjectID, StudentID)
(SubjectID)
(StudentID)
();
Review: The alternative (not provided for obvious reasons) is not just lengthy & repetitive (thus error-prone) but primarily not as performant as is otherwise possible.

MERGE INTO / UPSERT

Support: Yes (in 9.5)
Doable with alternative SQL? : Yes (for 9.4 and below)

Is the alternative as Performant?

No: This is because the alternative (URL given below) is a BEGIN/EXCEPTION based solution which is (time-wise) costly and an in-built support would certainly be faster.

Description: For those new to the complexity of MERGE (or UPSERT) please read this first.

TLDR: In the face of Concurrent Use, MERGE is difficult when a trying to balance Performance vs Integrity.

Unlike some other Database engines (that are sometimes okay with trading-off Integrity when it conflicts with Performance), PostgreSQL consistently prioritizes Data Integrity. The 'best' solution seems to have taken longer than expected, but considering that when a complicated open-source development model needs to coherently agree upon core feature additions, it really takes a few falling stars to get this piece-of-code in, with most people in support of it.
Example (SQL to create scenario + below SQL taken from here) MERGE INTO bonuses B

USING (

SELECT employee_id, salary

FROM employee

WHERE dept_no =20) E

ON (B.employee_id = E.employee_id)

WHEN MATCHED THEN

UPDATE SET B.bonus = E.salary * 0.1

WHEN NOT MATCHED THEN

INSERT (B.employee_id, B.bonus)
VALUES (E.employee_id, E.salary * 0.05);
Alternative: The PostgreSQL documentation mentions one recommended way of doing UPSERT / MERGE here. But again, this is non-performant and 9.5 based support for INSERT .. ON CONFLICT (a.k.a. UPSERT).
History:

MySQL / Oracle / MSSQL support this very well.
Long-pending requirement as per Wiki and now finally has made through!