I See Continuous Improvement

I gave my second talk at droidcon dubai this year with my coworker, Oubai Abbasi. We spoke about the importance of continuous integration on Android, along with some of the nice things that it can help catch and that you can do with it. Here is a link to the speakerdeck.

runtime generics in an erasure world

as we already know, generics in java are a compile time concept to help enforce type safety. during compilation, type erasure kicks in, resulting in the underlying bytecode being free of any generics information.

sometimes, however, we need generics information at runtime (such as when we need to convert a json string into its object form, for example). i was curious, how does this work given that types are erased at compile time? in other words, how does gson’s TypeToken class work?

in other words, why does doing this work when there are no generics at runtime (especially when, instead of String, the object type is a custom data object, for example?)

final Type typeToken = new TypeToken<List<String>>(){}.getType();
final String json = "[\"one\", \"two\"]";
final List<String> items = new Gson().fromJson(json, typeToken);

the tldr;

this great answer on StackOverflow answers the question nicely.

in summary, the java language spec specifies what the erased type of parameterized types, nested types, array types, and type variables is. it then says that “the erasure of every other type is the type itself.” TypeToken uses this fact to maintain generics information. as the TypeToken class’s javadoc says:

Forces clients to create a subclass of this class which enables retrieval the type information even at runtime.

stepping back

stepping back a bit, it’s pretty phenomenal seeing the effects of type erasure on bytecode directly. consider these two classes:

import java.util.List;

public class WithGenerics {
   List<String> data;


import java.util.List;

public class WithoutGenerics {
   List data;

if we compile these via javac and then look at the bytecode (using javap -v or using classyshark-bytecode-viewer), we’ll see:

notice that the bytecode is exactly the same for both classes. the only exception is that the type information is present in the signature of the WithGenerics class. if we are to run javap -v, we’ll see that this signature references the constant pool, where the type actually is.

  java.util.List<java.lang.String> data;
    descriptor: Ljava/util/List;
    Signature: #7    // Ljava/util/List<Ljava/lang/String;>;

in contrast, looking at WithoutGenerics, we’d see:

  java.util.List data;
    descriptor: Ljava/util/List;

“the erasure of every other type”

let’s take another example -

public class InnerType {
   public static class Internal<T> {}

   public static void main(String[] args) {

after running javac, we end up with two classes - InnerType.class and InnerType$Internal.class. looking at InnerType$Internal.class via javap -v, we see the class defined as:

public class InnerType$Internal<T extends java.lang.Object> extends java.lang.Object

if we try to display the class information like this:

public class InnerType {
   public static class Internal<T> {}

   public static void main(String[] args) {
      Internal<String> internal = new Internal<>();
      Class<?> classType = internal.getClass();

      System.out.println(classType + ", " + classType.getGenericSuperclass());

we get InnerType$Internal, with a superclass of java.lang.Object. now let’s try to modify the example slightly, and create an anonymous subclass of Internal, by doing this:

Internal<String> internal = new Internal<String>(){
   /* we could override methods here if we wanted to */

by just making that change, the app now writes that the class is InnerType$1, with a generic superclass of InnerType.InnerType$Internal<java.lang.String>. this generic superclass is actually a parameterized type, so we can cast it and extract extra information by doing something like this:

ParameterizedType t =
   (ParameterizedType) classType.getGenericSuperclass();
System.out.println(t.getOwnerType() + ", " + t.getRawType() + ", " +

if we run this, we now get an owner type of InnerType, a raw type of InnerType$Internal, and the actual type arguments of java.lang.String.

what about TypeToken?

if we look back at the first Gson example, we notice the use of a TypeToken class provided by Gson. what does this class do? we care about two classes here, TypeToken, and $Gson$Types. looking at the constructor for TypeToken, we can see it does 3 things:

  1. calls a canonicalize method on the type
  2. gets the raw type
  3. calculates a hashcode

most importantly, the canonicalize method exists in $Gson$Types and returns a specific Type depending on the actual Type passed in - if it’s an array, for example, a GenericArrayTypeImpl is made. in the example above, a ParameterizedTypeImpl would be made, using the owner type, the raw type, and the actual arguments.

in this case, as callers of Gson’s api, we make a new TypeToken with our generic type parameters. internally, this generates a ParameterizedTypeImpl that can then be used within Gson to do the right thing during deserialization.


in summary, whereas erasure erases generic types at compile time, libraries like gson take advantage of the fact that some types erase to themselves to have access to the generic type at runtime.

3 things i learned about rxjava 2 subjects


sometime in late december or early january, i decided to write a blog post per month. since it’s january 31st, i figured i should write about something to avoid dropping the ball so early in the year.

i introduced one of my friends to rxjava 2 not too long ago - his initial reaction was, “what? why would i want this?” - a few days later, it turned to, “hey, this is pretty cool!” - a few days after that, i learned several things from him as he ran into issues while migrating parts of his code.

consequently, i wanted to share these things, as they weren’t perfectly obvious to me (though in retrospect, they perhaps should have been).

also, i do realize that many consider subjects as “the mutable state of the reactive world” that should be avoided, but sometimes, they are a pretty good tool to use, especially when an entire code base is not yet reactive.

retry doesn’t actually retry

consider this code:

  public void testRetry() {
    TestObserver<Integer> ts = new TestObserver<>();

    final boolean[] val = new boolean[] { true };
    Observable.just(1, 2, 3)
        .map(i -> {
          // fail only the first time that i = 2
          if (i == 2 && val[0]) {
            val[0] = false;
            throw new IllegalArgumentException("throw!");
          return i;

    assertThat(ts.values()).containsAllOf(1, 2, 3);

this test passes, and the values emitted are [1, 1, 2, 3]. i’ll come back to the repetition in a tiny bit. now consider if we make a small change to this code, so that it uses a subject instead:

  public void testRetrySubject() {
    TestObserver<Integer> ts = new TestObserver<>();
    Subject<Integer> subject = PublishSubject.create();

    final boolean[] val = new boolean[] { true };
        .map(i -> {
          if (i == 2 && val[0]) {
            val[0] = false;
            throw new IllegalArgumentException("throw!");
          return i;


    assertThat(ts.values()).containsAllOf(1, 2, 3);

this test now fails, outputting only [1, 3]. so, why is this the case? the reason is that retry doesn’t actually retry anything - it just resubscribes to the source observable.

so if we think of the flow in this case, we call onNext with 1, which is observed by the subscriber. we then call it with 2, which fails because we throw an exception, and causes us to resubscribe. resubscribing to it doesn’t cause anything to emit. when 3 is passed in, we then observe it.

we can prove this by replacing the PublishSubject with a BehaviorSubject - doing so will result in [1, 2, 3] (because the subject caches the last onNext value it received, which was 2, so it gets replayed upon resubscribing).

note that the fact that retry resubscribes to the source observable is also why data can be repeated (as seen in the first example without subjects) - so when 2 fails the first time around, we re-subscribe, and thus get an onNext of 1, 2, and then 3, thus resulting in the repetition of the 1.

calling subscribeOn on a subject has no practical effect

this one was strange to me at first, but made sense once i realized that the reasoning for this was the same as that of why only the subscribeOn closest to the source matters - in summary, it’s because once onNext is called, downstream will receive the value on the same thread onNext was called on. thus, if you have subject.subscribeOn(Schedulers.io()), but call subject.onNext(value) from the main thread, the downstream will receive onNext on the main thread.

see also what i wrote here in the section about “only the subscribeOn closest to the source matters” (while that article was about rx1, it’s still relevant in rx2).

observeOn’s buffer and rebatchRequests

this comes from my friend’s question on stackoverflow. suppose we have a case where ui events trigger some work, but we only want to do that work if it’s not currently already being done - an example of this is a button that does work - and let’s suppose this takes a good amount of time, during which someone can click the button a few more times. if the work is being done, we don’t want to restart it, but if no work is being done, we can start doing the work.

my friend realized, “aha, this sounds like something that backpressure can solve, let me use BackpressureStrategy.LATEST!” - and so he implemented his solution to look something like this:

  public void testWork() {
    TestSubscriber<Integer> ts = new TestSubscriber<>();
    Subject<Integer> subject = PublishSubject.create();

        .map(ignored -> {
          System.err.println("before work");
          return ignored;
        .map(ignored -> {
          return ignored;

    for (int i = 0; i < 32; i++) {


this ended up failing, running the work 32 times - once for each and every emission of the subject. why?

as i learned from David’s answer, this is because observeOn has a buffer. since BackpressureStrategy.LATEST only keeps the latest value “if the downstream can’t keep up,” and since the default buffer size is 128 (unless it is overwritten by a system preference, in which case it must at least be 16), all the onNexts will be placed in a buffer until they can be sent downstream. in other words, backpressure doesn’t take effect here.

one solution i came up with based on this was replacing the first observeOn with observeOn(Schedulers.io(), false, 1) - this observeOn is called by the standard observeOn, with false for delayError, and bufferSize() for the buffer size. doing this results in the work only being done twice instead of 32 times. David said this would work, but would result in “time snapshots” as opposed to the latest event being processed (because as the worker was being processed, item 2 would be in observeOn’s queue, and would be sent downstream after the worker finishes - anything after 2 would be dropped until 2 is sent downstream).

David’s solution that actually gives you the latest was interesting - first, he used delay with 0ms as a means of switching threads without a buffer (i.e. .delay(0, TimeUnit.MILLISECONDS, Schedulers.computation())). then, he calls .rebatchRequests(1).

rebatchRequests was added as an experimental operator in RxJava 1, and this was the first time i had seen it. from my understanding, this is like a valve of sorts - it requests n items from upstream (based on the parameter passed in to it) - once 75% of them have been emitted downstream, it will request another n items from upstream.

it’s easier to understand what this is really doing when we look at how it’s implemented -

    public final Flowable<T> rebatchRequests(int n) {
        return observeOn(ImmediateThinScheduler.INSTANCE, true, n);

hey, cool! it’s calling observeOn with a buffer size of n, which, in this case, is 1. ImmediateThinScheduler is a scheduler that runs things immediately on the current thread.

haramain updates

it’s been a while since i first wrote about Haramain. there have been some pretty massive updates recently that i wanted to briefly write about here.

an app for iOS

i’ve wanted to release an app for iOS for a long time now, and unfortunately never got around to doing so. i thought Haramain was a good candidate to be a first app, and so august 29th of this year, i finally released Haramain for iOS, which is my first iOS app (i’ve played with iOS before on many occasions, but this is the first app that i wrote end to end).

the iOS app features a new design by Ahmed Galal and most of the features found on Android (except autoplay, which insha’Allah should come in the future).

android updates

so the Android app also hasn’t seen updates for a while now, and with the iOS app out and with a new design, it was time to bring the new design to Android, with some modifications.

so today, i released an update to Haramain for Android. this new version features Ahmad’s new design. it also adds Chromecast support and support for Android 7.1 shortcuts.

future plans

insha’Allah in the future, we have a few other things planned - namely search, Arabic support, and lists of favorites. stay tuned!

rxjava and a synchronous main thread

not too long ago, someone pointed me to this blog post about keeping your main thread synchronous. it referenced Ray Ryan’s excellent talk about the matter. the talk and blog post lead me to a set of investigations, which prompted me to write this blog post.

quick note: why keep the main thread synchronous

if you are curious as to the kinds of problems that can occur if the main thread isn’t synchronous (or why it’s not synchronous when you use observeOn), see this post about the main thread for a good explanation.

the summary is that everytime you use handler.post(), you post something to be run later, and you don’t have any guarantees as to when it will be run. a common case is when you schedule something to update the ui, but before it actually runs, an onDestroy comes in, causing the code to update the ui after the destruction of the activity.

in rx, specifically, observeOn(AndroidSchedulers.mainThread()) causes a handler.sendMessageDelayed (see LooperScheduler.java), which could cause code to run at a point after we are thought to have unsubscribed, thus causing issues.

basic rules of rxjava threading

in many of the talks about rxjava1, we find a set of repeated rules about rxjava and threading:

  • rxjava is single threaded by default unless you explicitly ask it otherwise
  • subscribeOn only affects upstream
  • only the subscribeOn closest to the source matters
  • observeOn only affects downstream

while these are all true, there are a few minor points that were not immediately obvious to me early on about the first three, so i would like to elaborate a bit on those.

rxjava is single threaded by default

as long as you do not use observeOn, subscribeOn, or an operator that runs on a particular scheduler (ex timer), the callback will be receieved on the thread subscribe happened on.

subscribeOn only affects upstream

one subtle point - consider:

Observable.just(1, 2, 3)
    .subscribe(integer -> {
      Log.d(TAG, "got value on " + Thread.currentThread().getName());

despite subscribeOn only affecting upstream, this will always print the result on an io thread, irrespective of the thread on which we called this code. this is because subscribeOn subscribes to the observable on the thread passed in, which means that onNext will be called on that particular thread.

only the subscribeOn closest to the source matters

consider this code:

Observable.just(1, 2, 3)
    .doOnSubscribe(() -> 
        Log.d(TAG, "subscribe to just on " +
    .filter(integer -> integer % 2 == 0)
    .doOnSubscribe(() ->
        Log.d(TAG, "subscribe to filter on " +
    .subscribe(integer -> {
      Log.d(TAG, "got value on " + Thread.currentThread().getName());

running this example results in:

D/are: subscribe to filter on RxComputationScheduler-1
D/are: subscribe to just on RxIoScheduler-2
D/are: got value on RxIoScheduler-2

if, however, we changed doOnSubscribe with doOnNext in the code block above, we’d instead get:

D/are: onNext from just with RxIoScheduler-2
D/are: onNext from just with RxIoScheduler-2
D/are: onNext from filter with RxIoScheduler-2
D/are: got value on RxIoScheduler-2
D/are: onNext from just with RxIoScheduler-2

the caveat here is that the subscribeOn closest to the source is the one that determines which thread onNext will get called on (but subscriptions still happen on the thread specified by subscribeOn).

the reason for this is that each subscribeOn subscribes to the upstream observable on that particular thread.

let’s take an example - given:


let’s break down what happens:


  • Observable.just(1,2,3) internally calls Observable.create(new OnSubscribeFromArray()), so we have an Observable which will call OnSubscribeFromArray on subscribe.

  • we then call subscribeOn(Schedulers.io()), which calls Observable.create(new OperatorSubscribeOn(this, scheduler)), where this is the Observable from above, and scheduler is io. In other words, we now have an Observable, which has an onSubscribe that will subscribe to the Observable from the step above on the io thread.

  • we then call subscribeOn(Schedulers.computation()), which calls Observable.create(new OperatorSubscribeOn(this, scheduler)), where this is the Observable from above, and scheduler is computation.

  • finally, we call subscribe, which calls Observable.subscribe, which calls the method from OperatorSubscribeOn from point 3 - this subscribes to the Observable from point 2 on the computation thread. Ultimately, this causes the OperatorSubscribeOn from point 2 to be called, which then calls subscribe to the Observable from point 1 on the io thread. OnSubscribeFromArray produces values on the same thread, thus causing all the items to be emitted on the io thread.

see the source for subscribeOn for more details.

running things in parallel with rxjava

in order to run things in parallel, we use flatMap or concatMap, with multiple observers that can then subscribeOn whatever scheduler they want to. the difference between concatMap and flatMap is that flatMap can emit items out of order, whereas concatMap will always emit items in order.

so what does this do? flatMap is essentially a merge, which “combines multiple Observables into one by merging their emissions”2. note that the observable contract stipulates that “Observables must issue notifications to observers serially (not in parallel).” this means that onNext will not be called concurrently, and part of merge’s job is to make sure that onNext is only called by one thread at a time.

for more on this, see Thomas Nield’s article about achieving parallelization, and also, see David Karnok’s article about FlatMap.

special thanks to Michael Evans for proofreading this.

  1. both of the aforementioned talks are definitely worth watching if you haven’t already seen them! ↩︎

  2. quote from rx merge docs ↩︎

rtl on Android

i gave my first talk at droidcon sf this year. the talk was about RTL on Android. check it out if you haven’t already (the slides are here).

on LinearLayout measures

seeing that LinearLayout is one of the most often used ViewGroup types in Android development, i wanted to write a little bit about how LinearLayout measures its children (this was also inspired by Sriram’s post about Custom ViewGroups).

tldr; (key takeaways that this post will discuss):

  1. a (non-nested) LinearLayout will measure each non-hidden child either 1x, 2x, or 3x (gone views aren’t measured).
  2. whenever possible, set android:baselineAligned to false, especially when using weights.
  3. in a horizontal LinearLayout with wrap_content height, avoid setting any of the children’s heights to match_parent. conversely, in a vertical LinearLayout with wrap_content width, avoid setting any of the children’s widths to match_parent.

LinearLayout and child measurement

a (non-nested) LinearLayout will end up measuring each non-hidden view either 1x, 2x, or 3x. here are the rules:

  1. a View with android:visibility="gone" won’t be measured.
  2. a View will default to being measured once, unless more measurements are necessary (see rules 3 and 4).
  3. when the non-android:orientation direction of the LinearLayout is wrap_content and its child’s is match_parent, the child will be measured an extra time.
    • in a horizontal LinearLayout with wrap_content height, a child with match_parent height will be measured an extra time.
    • in a vertical LinearLayout with wrap_content width, a child with match_parent width will be measured an extra time.
  4. a child with a non-zero android:layout_weight will be measured an extra time if:
    • in a horizontal LinearLayout with non-wrap_content width, android:layout_width is not 0dp, (or is 0dp, but the LinearLayout doesn’t have android:baselineAligned="false" set).
    • in a vertical LinearLayout with non-wrap_content height, if android:layout_height is not 0dp.

the first two rules are fairly straight forward - for optimization purposes, a view that is set to gone will not be measured. secondly, every view must be measured at least once, so the default is that each view will be measured once, unless something causes it to have to measure more.

the width/height mismatch measure

consider a horizontal LinearLayout with two TextView children.

if the height of this LinearLayout is match_parent or a fixed value (ex 32dp), then measuring the height of the children is easy:

  • if the child has a fixed height, use that
  • if the child’s height is match_parent, then pass in the LinearLayout’s height with a MeasureSpec mode of EXACTLY.
  • if the child’s height is wrap_content, then pass in the LinearLayout’s height with a MeasureSpec mode of AT_MOST.

but what if the LinearLayout’s height is wrap_content? in this case, measuring the child is sometimes easy:

  • if the child has a fixed height, use that.
  • if the child has a height of wrap_content, pass a height MeasureSpec with a mode of AT_MOST and the available space to the LinearLayout.

if, however, the child has a height of match_parent, what we’re really saying is, “i want this view to have the same height as the LinearLayout.”

so what is the height of the LinearLayout? in the case where the LinearLayout’s height is set to wrap_content, the height is not known until each child has been measured (the height MeasureSpec received in onMeasure by the LinearLayout itself will be AT_MOST with however much space can be taken up by it).

consequently, children with a match_parent height also have to wait until the LinearLayout’s height is determined, and then measured again in accordance to that height.

note that the same is true for vertical LinearLayouts when they have wrap_content widths and children with match_parent widths.

(note that in LinearLayout.java, you’ll see forceUniformHeight and forceUniformWidth in measureHorizontal and measureVertical, respectively, being called at the end of the measure cycle when they need to be).


here’s an example that shows this effect - compare:

the right square in the first image is measured twice, because its height is set to match_parent when the LinearLayout’s height is set to wrap_content. if these match (ex both are wrap_content, both are match_parent, or one of them is fixed), we eliminate this problem1.

note that sometimes, you really have to use match_parent - for example, if the right side had a background and we wanted it to be exactly the same height as the left side, we have no choice but to use match_parent for the height (unless we can specify a fixed height or change the parent, etc).


setting a non-zero android:layout_weight may cause a View to be measured an extra time.

first, let’s consider when a non-zero android:layout_weight does not cause an extra measure pass -

for a vertical LinearLayout:

  • when the LinearLayout has a layout_height of wrap_content, there is no extra measure (this makes sense, since there is no “extra space” to divvy up amongst the views at the end, since we’re telling the LinearLayout to only be as large as its content).
  • when the layout_height of the child is 0dp.

for a horizontal LinearLayout:

  • when the LinearLayout has a layout_width of wrap_content, there is no extra measure.
  • when the layout_width of the child is 0dp, and the LinearLayout itself has android:baselineAligned="false" set.

in all other cases, the view with non-zero layout_weight will be measured an extra time.


the documentation for android:baselineAligned says:

When set to false, prevents the layout from aligning its children’s baselines. This attribute is particularly useful when the children use different values for gravity. The default value is true.

this is generally useful for aligning text with the same gravity. consider:

and compare it to:

here’s a more visual representation:

so when can you turn it off?

  1. if the children of the LinearLayout don’t need to be baseline aligned (i.e. are not TextViews or ImageViews with setBaseline or setBaselineAlignedBottom set, etc).
  2. if the children are TextViews but all have different alignments anyway.
  3. if it doesn’t matter (for example in the above screenshots, because both sides use English text with the same text size, the images are identical - if they were different languages, or if the text size of one was different than the other, the two would definitely be different).

here’s a visual representation of how baselineAlignment can make a difference:

why does this matter?

why write a blog post about this, and why does it matter? there are 2 main reasons - first, given the popularity of LinearLayout within most apps, its important to know how it works to have great performing apps.

secondly, measures can be expensive - and avoiding extra work is always great! this is especially relevant when you consider nested LinearLayouts - let’s suppose that a nested LinearLayout will be measured 3 times, and has children that it would ordinarily measure 3x - those children now get measured 72 times!

  1. note that this is only true for a non-nested LinearLayout - if instead, the aforementioned layout is the child of a vertical LinearLayout with a height of wrap_content, the TextView with match_parent height will still get measured twice, because the extra measure at the end happens when a child is match_parent and the parent’s height MeasureSpec is not EXACTLY (in this case, its AT_MOST due to wrap_content). ↩︎

  2. in theory, they’d get measured 9 times, but the variation in the specs that are passed in to the nested LinearLayout let some of the views skip extra measures, especially in the first 2 measure passes.

intellij structural replace

intellij’s structural search and replace is an amazing feature that can save a lot of time. the value of this feature first hit home in the excellent Android Studio for Experts talk during Android Dev Summit 2015 (excellent video if you haven’t already seen it, btw!)

today, i was trying to replace my usages of android.util.Log.* with Timber. to use Timber, we’d need to manually replace Log.d(TAG, "message"); with Timber.d("message");, and the same for Log.e, Log.w, and so on. there’s also a version of Log.* that takes in an exception as a paramter that we have to keep in mind as well.

sounds like a perfect job for structural search - we bring it up by using ⌘ + shift + A to search for actions. type Structurally, and choose Replace Structurally.

we’ll get a dialog that we’ll fill out like this:

so to do this, set the search template to:

android.util.Log.$something$($tag$, $params$);

set the replacement template to:


note that the fully qualified domain name in the replacement ensures that the import is handled for you, as long as shorten fully qualified names is checked.

finally, choose Edit variables, choose params on the left, and click the unlimited box by maximum count. this is so that we match both Log.d(TAG, "message"); and Log.d(TAG, "message", exception);.

some notes about the options:

  • shorten fully qualified names replaces things like android.util.Log.d with Log.d.
  • reformat according to style will fix the indentation if it’s wrong.
  • use static import if possible will prefer import static - i’ve unchecked this because i’ve found that it did this for many things that i didn’t want it done for.

and there you have it. as always, source control is your friend, so make sure to do this on a branch where you can validate the changes and easily roll back if things aren’t quite right.

quran android no longer open source

update - as of 12/31/2015, i’ve re-open sourced the app after the discussion here.

today is a very sad day for me, since i have finally decided to close quran android’s source code once and for all. throughout the past few years, i’ve come across several examples of people republishing the app with the intention of making money (placing ads on the index page, or, worse yet, on the top and bottom of each quran page while reading). recently, someone brought to my attention an example of someone repackaging the apk and shipping it with malware.

for example, fabric shows me this crash:

this issue is happening on the current version in the play store, which was pushed on 5/31/2015. looking at the graph, you can see that until mid July, there were no crashes (because this sdk doesn’t exist in our app). after mid-July, someone pushed this (modified) apk somewhere, people started downloading it, and it started reporting crashes to me.

a similar thing happened some time back, but with the person pushing their own version:

a few years ago, i closed the source to quran android for similar reasons, only to end up bringing it back after a few months, figuring the benefit of it being open outweighs the harm. unfortunately, this time, the decision is not meant to be reversed insha’Allah.

i’ve come to the difficult realization that apps are not meant to be open sourced, because, when you open source it, there will always be people who try to profit off of your work (or destroy its credibility, etc).

quran will, insha’Allah, continue to be developed, but it won’t be open source anymore. the existing github project will be used to track issues. while i understand that removing the code will not completely stop these things from happening (apks are very easy to reverse engineer, and people with a will to do something will find a way), at the very least, i can stop making it super easy for people.

update - this seems to be a trojan that affects multiple apps downloaded from unofficial app stores.

migrating to hugo from hexo

i haven’t written in a long time. it’s not that i have nothing to write about (i actually have a set of post ideas written somewhere), but more of just not having the time for writing.

just yesterday, after seeing a fellow developer’s new blog based on jekyll, i decided to search for a nicer theme for this blog. in the process, i came across hugo, a blogging engine written using go. the two things that interested me the most about it was the fact that it was supposed to be super fast (whereas regenerating files for hexo still took a bit of time), that it was written in go (which i hope to learn one day), and that i found a theme i liked for it. the fact that chrome on the latest el capitan beta doesn’t load my site at all was yet another reason to push me to go for it.

the process didn’t end up being too bad, and i’ll document some interesting tidbits i found here in case someone else goes through the migration in the future.

fixing the metadata.

there are a few important pieces required to fix the metadata - first, ensuring all lines start with ---. for me, any files i generated on hexo didn’t have --- on the top of them. secondly, fixing the dates. hexo dates were in a different format than the iso8601 format dates. furthermore, some of the entries i had wrapped their dates with single quotes.

i found something here to do this, and modified it to fix the single quote issue as well:

# ensure dates don't start with single quotes
for file in *; do awk '{
if ($1 == "date:") {
  gsub("\047", "", $0); print;
} else {
  print $0;
}' "$file" > temp.md && mv temp.md $file ; done

# fix the dates and add the three dashes as the first line
for file in *; do awk '{
  if (NR == 1) { print "---"; }
  if ($1 == "date:") {
    printf("%s %sT%s-05:00\n", $1, $2, $3);
  } else {
    print $0;
}' "$file" > temp.md && mv temp.md $file ; done

# wrap dates with quotes that aren't wrapped in quotes
for file in *; do awk '{
  if ($1 == "date:") {
    if ($2 ~ /^"/) {
      print $0;
    } else {
      printf("%s \"%s\"\n", $1, $2);
  } else { print $0; }
}' "$file" > temp.md && mv temp.md $file; done

one last thing was that some of the entries i had were missing the seconds field, and hugo complained about being unable to parse their times. since they were only a handfull of entries, i went ahead and fixed them manually.

fixing code highlighting

i used something like this to convert the codeblock style to the pygments driven code highlighting:

for file in *.md; do awk '{
if ($2 == "codeblock") { gsub("lang:", "", $(NF-1));
  printf("%s< highlight %s >}}\n", "{{", $(NF-1));
} else if ($2 == "endcodeblock") {
  printf("%s< /highlight >}}\n", "{{");
} else { print $0; }
}' "$file" > temp.md && mv temp.md $file ; done

as a recommendation, if trying to use the hyde theme, set pygmentsUseClasses to true in the configuration file.

after doing this, i realized that i wanted to use hyde-x, which by default uses highlight.js. i made this work (after running the above) by running:

find . -exec sed -i '.bak' 's/{{< highlight \(.*\) >}}/```\1/' {} \;
find . -exec sed -i '.bak' 's/{{< \/highlight >}}/```/' {} \;

fixing images

hugo supports shortcodes as a means for extending markdown. using a combination of the migrate to hugo from jekyll article, the shortcodes documentation, and migrating to hugo from octopress, i came up with a similar approach for fixing images that had a width and height in them.

find . -exec sed -i '.bak' 's/{% img \([^ ]*\) \([^ ]*\) \([^ ]*\) %}/{{< img src="\1" width="\2" height="\3" >}}/' {} \;

this was combined with adding an img shortcode, which looked like this:

<img src="{{ .Get "src" }}"
{{ if .Get "width" }} width="{{.Get "width" }}" {{ end }}
{{ if .Get "height" }} height="{{.Get "height" }}" {{ end }} >

in addition, i modified the css to remove display:block from poole.css under images to allow having two images side by side.

one other minor thing i found was that layout: post, which was either an artifact of hexo or octopress, was breaking disqus rendering. i fixed this by doing this:

for file in *.md; do
   grep -v "layout: post" $file > temp.md
   mv temp.md $file

a little bit of extra tweaking to the settings and layouts and i was good to go! hopefully, this will push me to write posts a little bit more frequently :)