A little adventure: porting from J2ME to iPhone

Three weeks ago, I began porting two of my J2ME applications, the Catholic Mobile Prayerbook and the Way of the Cross, into iPhone applications.

Milestone:
Week 1: Port to Objective C (and HTML!) - easy & fast
Week 2: Test on simulators and friends' iPhones - oh no, why won't it run? And there are NO error messages?
Week 3: Got my hands on an iPhone - took a day or two to get it to run a Release version

Status Now: Uploading to iTunes' App Store ... waiting for review

Some useful links when you are on the last stage, about to submit your application to the iTunes' App Store:

1. "The binary you uploaded was invalid. The signature was invalid, or it was not signed with an Apple submission certificate."
http://discussions.apple.com/thread.jspa?threadID=1590980&tstart=0

2. Two different days, project unchanged, different errors?
http://discussions.apple.com/message.jspa?messageID=9167082#9167082
http://blog.rightsprite.com/2008/11/iphone-buildingtesting-for-device.html

3. OK... all done? Now for the money, wait a minute.. I'm not an American taxpayer; what are all these tax numbers for??!!
http://www.opensoftwarepartners.com/2009/filling-out-apples-manage-your-tax-information-for-app-store-ireland/
http://discussions.apple.com/thread.jspa?messageID=9191027

Hope it helps someone out there ;)

Hot off the 'oven'

I'm raising funds to go to grad school this Fall. I'm putting up "Catholic Mobile Prayerbook" - available for download here.

What is Catholic Mobile Prayerbook?

It is a small mobile application containing the common basic prayers that Catholics commonly say. From your all-time favorite "Our Father" to various devotions you can think of.

Why would I need Catholic Mobile Prayerbook?

For those times that you have to wait, it is a handy aide to help you pray. It'd be useful also when you forget the Act of Contrition in the middle of the confessional booth. Also, when you are asked to say grace before meal. Or to lead any prayer. Wherever. Whenever. One doesn't need a reason to pray.

What prayers are available inside?

The basics: Our Father, Hail Mary, Glory Be.

Devotions to Our Lady: Memorare, Angelus, Hail Holy Queen, Rosary

Daily Prayers: Morning Offering, Spiritual Communion, Apostles' Creed, Prayers before & after meal, Thanksgiving

Special Prayers: Act of Contrition, Adoro te Devote, Prayers for the pope, Te Deum, Prayer to St Michael the Archangel.

To request for more prayers to be made available, write me

What languages is it available in?

English, Español (Spanish), Latin, and Bahasa Indonesia.

Coming Soon: Francais, Italiano and Deutsch.

Does Prayerbook support my phone?

Prayerbook is developed on Java ME Platform. Nearly all recent mobile phones support Java ME. Check here to make sure.

How do I install it?

1. If you click the download link below, it'll take you to a payment page, and then ask you for your phone number.


2. Enter your cell phone number as requested: Your country code, followed by your cell phone number.


3. Eg: 441223472777 (if you live in the UK), or 6598765432 (if you live in Singapore).


4. As soon as you enter your phone number, an SMS is sent with a link to the file.


5. Click on the link to download the file.


6. If permission is asked to open/run the file, do allow it: select "Yes".

That's it!

How big is it?

Depending on the version you download, it ranges between 60-80KB. That should translate to less than a dollar to download via your phone. If you'd like to save money on the download cost, download it to your computer and send it via Bluetooth!

It says installation was successful but I can't find it!

On different phones, Java applications are stored at different places. On most Nokia phone, click the Menu to see all the installed applications. Sometimes it will be shown under "Game", or "Application", or "Installations". On Samsung phone, do check out "Java World".

If you really can't figure it out, email me your phone brand and model, and the name of the file you downloaded.

How much does it cost?

A single-language version costs $1.99, a two-language version costs $3.00 and all-languages version costs $5.00.

To promote this amongst the Indonesian-speaking people, the Bahasa Indonesia version is now FREE!

Why do I have to pay?

As much as I'd really like to encourage people to pray, I need to raise some funds to go to grad school. So *PLEASE* support my fundraising effort! (The story of why I am going to grad school is topic for another post -- if you'd like to know, mail me)

For free download, I have made available a Rosary widget for Yahoo! Widget. Get it here.

I'm also available for any freelance work to develop web widgets, desktop widgts, mobile widgets, and mobile applications. Drop me a line if you'd like to help me put through grad school!

Download Section

To have me email it to you:

Single Language Version

English $1.99 Spanish (Espanol) $1.99 Latin $1.99 Bahasa Indonesia $1.99

Your Email (to which the app is sent):

Two-languages version

English-Espanol $3.00 English-Latin $3.00 English-Bahasa Indonesia $3.00 Espanol-Latin $3.00 Espanol-Bahasa Indonesia $3.00 Bahasa Indonesia-Latin $3.00

Alternative Email (in case your primary email doesn't work):

All-languages version (English, Español, Latin, Bahasa Indonesia)

Alternative Email (in case your primary email doesn't work):

 

To Download directly to your phone: (pay via your mobile phone bill)

Single Language (USD$1.99)

Español
English
Latin
Bahasa Indonesia (FREE)

Two Languages (USD$3.00)

English-Español
English-Latin
English-Bahasa Indonesia
Español-Latin
Español-Bahasa Indonesia
Bahasa Indonesia-Latin

All Languages (USD$5.00)

Multi-language

Learning Tip: document your steps!

These past few days I have been working on a little mobile app (J2ME / Java ME) that displays the various Catholic prayers and devotions. Simple stuffs like "Pater Noster" all the way to the mysteries of the rosary, and thanksgiving prayers, in multiple languages.

It started out pretty simply, and the complication happened only when I had to 'versionize' the source code and the resources used by different language versions. Some versions should be able to handle multiple-language requests from the user, some display only one language of choice.

Two problems: versioning and internationalization.

Internationalization has always been an interesting topic for most application developers because frankly, few of us bother to provision for Unicode display until the product is almost ready. Then began the scramble to make the application accommodate the havoc wreaked by the different encoding the input came in.

Versioning problems came about because I did not use CVS. Period. Well, there's more to the story than that, but bottomline is, there are existing tools that can track your files' changes.

During the course of this project, I have learned that Ant (which is the build tool I use to compile J2ME applications) is extremely nifty with multiple compilation paths. A little familiarity is required with the Ant build file, and more curiosity, to learn the various types of variable and property that are used to signal different customization and compilation paths.

For example (in J2ME Polish):

The element <sources> can be customized to point to different folders for different versions. So instead of re-naming your public classes, MyMIDlet_v1 or MyMIDlet_v2, you can stick with the same name in different folders.

The element <jarName> can contain references to other variables or properties, to be something like: "prayer-${polish.name}-${polish.locale}-${TwoLangCombi}.jar"

Those variables and properties can combine to indicate which resources folder you would like to use when compiling for a particular locale, or particular screen size, or particular handset brand.

Why am I writing this?
I have suffered greatly because I did not learn to use these 'best practices' in my daily work! I hope anyone of you who happens to read this and understand may benefit from my suffering and this little note ;)

J2ME Polish: garden-variety bugs

I'm writing this short collection of bugs I've encountered when working with J2ME Polish (RC4) on Samsung devices, just for my personal future reference & to help anyone who found this in passing:

1. The use of 'viewtype' midp2 (which in my case allows more flexible layout of elements) causes, in some devices, notably SGH-E900 and SGH-X820, the screen to freeze when any joystick key (up/down/right/left) is pressed.
   Workaround: Pressing the down key returns -1 as the index of the next element. Either implement your own extension of the MIDP2LayoutView class to correct this problem, or don't use it—use other view types.
   
   
2. Title bars sometimes are not rendered in a Form. Again, not all devices are affected; title bar does get rendered on Z230, Z400, and Z720 but NOT on E900 and X820. Might have something to do with Canvas sizes and Screen sizes, and/or fullscreen issues.
   Workaround: not known. More research needed.
   
   
3. On a Form, when implementing an ImageItem to act as a button, J2ME Polish may sometimes throw an error that the Command attached to the ImageItem cannot be executed. Again, the above devices seem to be the only ones affected.
   Workaround: The Command needs to be attached to the ImageItem via setDefaultCommand(...)
   

That's about it for now; this will be updated as more issues are encountered.

J2ME Woes

Have I ever mentioned I hate mobile phones? Yes, that's right. It's helped revolutionize our lives, I'm sure, but it's been a source of misery for the lot of developers who has the fortune (or mis-fortune) to attempt development of "write once, run everywhere" (i.e. the motto of Java programming). Not to mention I have been using so many test phones that I am nearly schizophrenic when it comes to phone keypads and most certainly paranoid when calls from unknown numbers (unknown, because the more recent contacts were not in the SIM card) arrived...

Before going further, it must be said that this is a late afternoon rant after a few days of being entirely submerged in the exasperating world of standard-less mobile devices. The landscape of software development for mobile devices today, looks a little like pre-DOS era of personal computers. Sloppy & non-compliant implementations of JVM on the mobile litter the mobile market.

Some J2ME Polish bugs encountered, either fixed or evaded:

losesFullScreen: an issue that affects Samsung SGH-D500, in which the screen rendered on the phone is cut-off at the top & the bottom. The catch is, using J2ME Polish, one has less control over the direct Canvas implementation. Hence even my Forms ended up getting cut.
Status: UNFIXED, left to the manufacturers to fix their device, or better, the mercy of God!

SoftKeyMappedToFire: an issue that affects the navigation keys of many Samsung devices--sometimes pressing the soft key results in the default command being selected. If the default command for that item is not defined, good luck to you :) Anything from screen freezing to a Samsung version of BSOD is likely.

MIDP2LayoutView: Pressing any key on any affected screen throws an ArrayIndexOutOfBounds exception. Initially I tried what the mailing list suggests, that is, to override J2ME Polish' MIDP2LayoutView class, but ran into further trouble trying to reference this class from the secondary compilation of StyleSheet. I gave up on this, threw "midp2" view out of the window.

There, here's a fulfilment of my quota at being Good Samaritan this week! (in view that most of this blog's visitors don't come here to read my spiritual blurbs but to read HOWTOs and battle stories about J2ME & Linux) If there's any bright lining in the dark cloud of J2ME woes, it'd be this maxim I've since taken personally: "Write once, pray it will work everywhere."

EXIF extraction for J2ME

I've been struggling with creating thumbnails in J2ME—not that it's hard to do—but given the limited heapsize and mobile handsets' processor capability, the traditional Java (J2SE) thumbnail-generation process fails when the picture whose thumbnail about to be generated is HUGE (eg. those created by >2 megapixel cameraphones).

In general, there are few approaches:

1. generate your own thumbnail (good for smaller JPEG pictures & PNG formats)
2. read (hidden?) thumbnail directories that some handsets create
3. try to extract thumbnail inside EXIF header of some JPEG pictures

The 2nd & 3rd options don't always work for all phones. Problem with #2 is that only on some handsets is the picture thumbnail generated into predetermined directories. For #3, some developers claim Nokia cameraphones do not generate JPEG formats with EXIF headers in it. Sony-Ericsson phones do, however!

So this is the original Java library that I used to extract the EXIF header and obtain the thumbnail offset, length and the thumbnail data itself. It needs porting, however, to be able to run on J2ME. I've ported it into J2ME and the (crude!) source for the test midlet is here. Some sample pictures can be found at the original site.

More information about EXIF tags can be found here, in case you'd like to tinker with more exciting information! If anyone comes to this blog by searching for this, please feel free to download it & use as you like. Original code is copyrighted and it belongs to Drew Noakes.

I hope this helps some people out there.

UPDATE! (Apr 1 '2009):
The domain where the source is hosted has expired ;) I have updated the link to point here: http://blvu.com/exif-j2me-src.zip

Good stuff

Sun Java Wireless Toolkit (WTK) v2.5 (beta) is out! Great APIs support include:

SIP API (for implementing IMS through SIP)
2D Vector Graphics API (if you know what a pain it is to resize images in J2ME, thank God for this!)
Advanced Multimedia Supplements API (for advanced multimedia stuffs, including advanced camera op)

Now, it's waiting time; time to wait for the handsets supporting these APIs to come into the market soon.

J2ME Bug: mapped SoftKey values

Description: In some Motorola handsets, J2ME application running Canvas does not seem to respond when the left, middle or right soft key is pressed.

Affects: Motorola V3, V3x, SLVR L6 (only tested on these)

Workaround: In your code, expect the values of left, middle, and right soft key press to be -21, -23, -22, instead of the original 21, 23, 22.

In J2ME Polish, change the affected devices specification in devices.xml to have these extra capability parameters as described here.

J2ME Polish: obfuscator bug

This bug has been the bane of my life for the past few weeks. To a non-IT person (that's a lot of you and I, before my company started), it might have seemed strange to want to obfuscate anything!

The American Heritage® Dictionary defines obfuscate as:
1. To make so confused or opaque as to be difficult to perceive or understand
2. To render indistinct or dim; darken

For people who gets irked when their code gets lifted, obfuscation can be done on their codebase. In J2ME, obfuscation helps to reduce the final JAR size by eliminating classes which have been declared but never referenced, at the same time renaming classes to one-letter or two-letter names, thus 'obfuscating' it for the code thief. It is a nearly standard practice, and I would like to document my little tête-à-tête (or a wrestle?) for others whose search take them here.

Description: J2ME Polish v1.2.4 seems to have a problem obfuscating code which contains the javax.microedition.media.Manager class.

Should we abandon Polish? What if your codebase has grown dependent on Polish? What if, worse, you've fallen in love with Polish' elegant UI? Both were true for me. So here's what I did:

1. Don't obfuscate
No go. I absolutely need two library JARs, one for floating-point-decoding operation (33KB), the other for xml parsing (kxml2-min.jar—18KB). Total JAR size was 295KB. Costs a bomb to download OTA. How can we ever hope anyone would download it to their phone?

2. Manually prune my code & optimize images
Eclipse helps me to see what methods and variables are not being used. I also renamed packages, methods, variables and packages to use shorter names while trying to remember their mnemonics! Basically, all the 'good programming practices' taught in school have to be thrown out of the window when it comes to making lean codes for mobile phones. I only have two 1-KB image files. My effort reduced the JAR size to 188KB.

3. Attempt to implement my UI using Canvas
This took me two days: euphoria and dypshoria took turns when it worked smoothly on Nokia 6230i and then I realized I might have to provision customization for every single phone model!

4. Attempt to fix it
I suspected this bug might be due to two JAR files used by J2ME Polish containing the Manager class in the same package javax.microedition.media. So I bravely combined the two JAR files and replaced the Manager class in Polish' JAR files with the original Sun Wireless Toolkit's. Didn't work.

5. Use Proguard's parameters in J2ME Polish
I tried introducing some parameters "-dontskipnonpubliclibraryclasses", "-dontusemixedcaseclassnames" and "-keep" on the Manager class, in the obfuscator element of the build.xml file. Didn't get past step one, because it requires me to overwrite a Polish obfuscator class.

6. Upgrade to J2ME Polish v1.3-beta4
Not only this version doesn't solve the problem, it introduces new bugs for the function Display.setCurrent(Alert, Displayable)

7. Use alternative obfuscators
Jode, or Retroguard both didn't work with v1.2.4 and v1.3-beta4.

8. Upgrade to ProGuard v3.4 or v3.6
Didn't work. These two versions are not backward-compatible with the calls that Polish v1.2.4 make.

9. Finally, (THIS WORKED!) Obfuscate outside J2ME Polish
I compiled the JARs in Polish for some fifty phone models without obfuscation. Open them using other Wireless Toolkits (Sun's, Siemens' or Sony Ericssons'), attach the source and the library JARs independently, rebuild with obfuscation (Proguard v3.2—same version as what Polish uses)! Smooth as butter. Extra patience required.

RESULT: Final JAR sizes are between 95KB to 106KB (depending on device models).

Talk about dismantling the whole building just to fix a little plumbing problem. What one wouldn't do for something one loves... Hope this helps, folks!

J2ME: Golden bug

I have exactly 13 days to release my company's app (and the other company's app too). After playing around with the look-n-feel, and running it hard to slim down the JAR file size, I finally got to look at the part that reads from the mobile phone's filesystem. At first, the bug's behavior is hardly noticeable underneath all those J2ME midlet permission pop-ups (one for every file listed).

Description:
On most Java MIDP2.0 phones supporting JSR-75 (FileConnection API), the following call should return the directory names requested:
System.getProperty("fileconn.dir.photos")
should return
"file:///c:/pictures/"(or something equivalent applicable to the handset make).
Similarly other getProperty calls to "fileconn.dir.tones", "fileconn.dir.videos", should return ringtones' and videos' respective folders.

On Sony Ericsson's models K750, W800 series and possibly its entire Java Platform 6 family, this call fails—returns nothing basically. The root cause of this was simple:

Cause
“fileconn.dir.xxx” has accidentally been misspelled as “filconn.dir.xxx” in those phones' Java API implementation.

Link here

Many programmers, especially those new to J2ME, spent countless hours of near despair wondering what on earth happened, or did not happen, on that tiny machine. Despair no more!

J2ME references

This week I've been grappling at low-level firefighting on the J2ME mobile platform. One of the most frustrating things I encountered is finding similar problem being reported in the various forums, but not the followup!

So here are some reference for those who stumble here through late-night Google search:

For newbies who get frustrated by JSR-135 non-compliant implementation in Nokia phones: theory vs. practice
Mobile Media API Support In Nokia Devices

For those who find their JARs too big / want to 'hide' their code so that codelifters have to work a little harder:
Obfuscators: Proguard, Retroguard, Jode, etc.

For those who want to 'polish' up their clunky app:
J2ME Polish—be warned, there may be some problem using J2ME Polish with Obfuscators!

Update: Will post more on my wrestling match with J2ME-Polish and Proguard later. Down and out for the week.

J2ME Midlet signing

I've been searching the 'Net for procedures and guidelines procuring a real digital certificate to sign mobile codes (in our case, Symbian .sis files and J2ME .jad & .jar files) before they get to our customers' mobile phones.

Not having much cash at hand, it's a little difficult for my company to pay a lot (between US$199-$500 annually) just for a piece of document (a digital one at that!) that basically ascertain you are who say you are on the 'Net. For some environments, it is becoming necessary to have a certificate before our code (application) will run at all. For others, like J2ME, it is simply to save all that hassle on the part of the users, who have to say "Yes" to every operation your app is meant to do on the mobile phones, such as taking picture, listing down their files, and connecting to the 'Net. What started out as a security requirement has become an obstacle to rolling out good, smooth user applications (who wants to say yes a hundred times just to display a hundred files?!)

In any case, here's summarizing what experience I've gathered from the people who've tried and sometimes succeeded, sometimes didn't:

1. Don't get your hopes up eliminating ALL those pesky pop-up permission windows
   
   Sometimes, not all the permissions requests you have indicated will be granted. Some handset firmwares/operators will relent enough to give you blanket permission; some will merely relent to ask the end-user's explicit permission only once at the beginning of the session. AFAIK, midlet signing is very useful when your app runs JSR-75 (so that your end users are spared having to click 'Yes' to every file/every contact in every folder that is about to be listed). You have been warned.
   
   
2. Expect different performance raise (or non-raise) on different manufacturers' (Nokia handsets should perform reasonably uniformly) and different handset models
   
   Same handsets in different countries and running different operators' SIM cards may end up giving you different behaviors. Some might 'repress' the permissions you requested. You have been warned a second time.
   
   
3. Make sure you get the right certificate: code-signing cert, not the SSL type!
   
   It might waste you a few hundred dollars if you get the wrong type. Also, not all Certificate Authorities (CAs) have their code-signing root certs already preloaded on the mobile handsets your app is meant to run on! Check and double-check! You have been warned a third time.
   
   

If you find this summary relevant but too short, here are tutorials & troubleshooting and helpful links to better, longer discussions on midlet signing issues. And I wish you success in your applications' launch!

J2ME: Write once, run anywhere?

Write once, pray it will run in more than one family of phones... that's more like it. Just collecting another J2ME anecdote that would have been funny if it's not so exasperating.

On different handsets, VideoControl has a getSnapshot(null) method that returns images of different formats. According to documentations, getSnapshot(null) returns a PNG format. Yet this returned PNG in Nokia Series60 handsets and JPG in Sony Ericsson handsets (K300i, K500i, K700i)! It might not seem like a big thing; but this means many many more lines of codes to handle different cases.

To think this escaped our notice for many months until it started crashing our black box libraries...??? Dear God, deliver us from bad API designers!

Part 4 of J2ME-Servlet mystery [SOLVED? I think so!]

Using the various methods of reading at the Servlet, I found that none of them worked perfectly:

1. With getReader() and reading from the raw InputStream, somehow the last 40bytes are always gone!

2. With getInputStream(), BufferedReader's readLine(), it gave me an ArrayIndexOutOfBoundsException when reading! The logs traced it back to a possible mismatch in the number of bytes indicated and what was actually found, in
sun.nio.cs.StreamDecoder$CharsetSD.readBytes() method

BUT

All these servlets are running on Tomcat 5.0.28. When I copied the servlet to another Tomcat server, this time v5.5.9, the servlet receives whatever J2ME client sends perfectly!

Is this a bug? This is a possibility I haven't entertained before, but it's worthwhile asking, seeing the number of similar questions out there.

*UPDATE*

This whole thing started as a series of posts of frustration, but they might contain clues to help you solve similar problem, as I wrote about the steps/approaches I've used to try solving the problem:

Part 3, Part 2, and Part 1

Warning: They're RANTS!

The illogic thickens..

3rd part of a frustrating technical rant:

I ran tcpdump on the server side, and what difference do I see between requests coming from J2ME client and a J2SE program?

The J2ME request has Transfer-Encoding: chunked at the header, and the body is cut into a few chunks, while the J2SE request is in one smooth body. Not that it should matter, because Tomcat 5.0.28 is supposed to be HTTP 1.1-compliant.

I didn't call OutputStream.flush(), but i suspect it'd have been called anyway because the request is more than 2048 bytes long.

It frustrates because I don't understand why it happens. Why can't Tomcat accept both requests and obtain the value for the getParameter(...) request in the same way? It's illogical. It defies common sense. What's next? Use StreamConnection? Just how does HttpServletRequest's getParameter(...) work anyway? How does it get parameters from a connection?

A few weird observations:
1. Content-Length != -1, despite the presence of chunking. Normally chunking would reset this header value.

2. A different source says 'do not set Content-Length' when using chunking... I wonder..

Unsere dame, Maria Knotenlöserin, Beten Sie für uns.

... spilleth over!

Today's earlier rant is spilling over to this post. It has been, what, perhaps a year since I last ranted about work in this blog (it's not meant to be a rant page...) But I've been really busy lately (both at work and troubled somewhat in the heart), and this latest thing just irks me to no end, not to mention it sucked a great many hours!

OK, so for you folks out there who by weird chance might just end up reading this post because Google archive it or something, here are some things to take note when using J2ME client to do a HTTP POST to a servlet sitting on a Tomcat container trying to read a parameter using getParameter(...)

Things which a lot of sample codes probably don't tell you upfront:

1. Order of calls is important
setRequestMethod(HttpConnection.POST), setRequestProperty("Content-Type", "application/x-www-form-urlencoded"), and all setRequestProperty(...) calls must be made before calling getOutputStream()!

2. Sometimes calling outputstream.flush() may result in chunking that some application servers cannot handle. Check yours. Use outputstream.close() to be safe.

3. setRequestProperty("Content-Type", "application/x-www-form-urlencoded") MUST be called.

4. If you don't have setRequestProperty("Content-Length", N) , the servlet side sometimes sees 2 requests-- header & body, rather than just 1 request. (Hint: watch HTTP traffic on your server side)

5. Lastly, of course, don't be silly, remember what you write to outputstream must be in the form of this regular expression: [yourParameterName=yourParameterValue]+ [&nextParameterName=nextParameterValue]*

And don't forget to pray to our Lady, Maria Knotenlöserin

Nonsense!

<start of rant>

For anyone who's struggled with J2ME-J2SE and servlet on Tomcat container, I wonder just how do you maintain your rational sanity when faced with problem like this:

URLEncoder.encode("param", "UTF-8")+ "=" + URLEncoder.encode(payload, "UTF-8") [WORKS!]

and

"param=" + URLEncoder.encode(payload, "UTF-8") [DOESN'T WORK!]

The doPost(...) method of the servlet is able to read "param" using request.getParameter("param") in the first method and not in the second method! Bloody nonsense!

And on top of this, both methods still could not work when the call to the servlet is made from a J2ME client. Woot!
<end of rant>

Seriously, I'm getting sick of this. Maybe I should study theology now.