Look at that, we’re approved!

Albeit a couple of weeks late, it’s time to post about how the Capstone semester wrapped up. The short version: good. But, a longer explanation is probably necessary.

Dyksen recruited our Mozilla team, along with the Urban Science team from Capstone as well, to demo our projects for a Strategic Partners Presentation back on the 19th. Despite some initial, how should I put this, terrifiedness, at presenting in front of corporate people, it ran pretty smoothly. Good idea from Brandon to pass the Surface tablet around so everyone could use the rotate gesture themselves. I guess it’s good when the first question asked after the demo is “Where can I get this?”

Of course, our project still wasn’t 100% finished yet. We did a good job of hiding our bugs at our alpha, beta, and Strategic Partners presentations, but with Design Day looming, it was time to rid our project of its remaining issues. My main problem was an incorrect animation transition when zooming past the edge of a web page. After ~9 weeks of tweaking and slowly fixing issues but getting a problem that the scrollbars can’t go past their maximum, I finally found the (rather obvious) answer: just set the translation origin to the corner where the web page is past the edge (instead of always top left by default). Then a new translation amount is only the number of pixels past the corner, which can transition back to zero for the animation (scrollbars don’t transition, so transition animation is done via translating the entire page. Of course, you can’t just leave the page like that, or it’ll be cut off when the user scrolls around, so once the animation is complete, THEN re-render with just scrollbars now that there’s no over-zooming past the edge. Simple, right?)

The other problem was that pesky canvas-jumping issue on the Macbook Pro (see previous posts for details). Still don’t know why it did that, but after finding what the problem was with the HD canvas idea (also see previous posts) and fixing that (I forgot to check if HD canvas is active before the if statement to decide whether or not to add a certain canvas to the stack – otherwise, sometimes that canvas got stuck there and wouldn’t go away after the gesture terminated). The new problem was that this HD canvas leaves a white background around the rest of the page – but that’s only visible to the user if they move the mouse while pinching, which pans over while zooming. Luckily, for the Macbook, the mouse is the trackpad itself, and while in a pinching gesture, the mouse doesn’t move – so, problem taken care of by itself. Of course, on the Mac or the Surface Pro, the mouse CAN move while pinching (the Surface Pro’s mouse cursor is just the midpoint between one’s 2 fingers, which moves while pinching, and there’s an external mouse for the Mac, so one can pinch on the trackpad with one hand and move the mouse with their other hand). But again, problem taken care of – this canvas-jumps issue doesn’t occur for whatever reason on the Mac or Surface. It’s still nice to see a higher-resolution image after zooming in to 200% the original size, but since that white border becomes noticeable once panning over, the canvas is only displayed IF the mouse hasn’t changed position. Somehow it all worked out…

So, with a few days before Design Day, we were happy enough with our add-on that it came time for submission. And lo-and-behold, a couple of weeks later (May 4), turns out it got accepted! It can now be downloaded by anyone at https://addons.mozilla.org/en-US/firefox/addon/pinch-to-zoom-and-double-ta/
(Why the link ends in “double-ta” instead of “double-tap” I don’t know. Probably a missed letter while copy-and-pasting)

Not to say the add-on was perfect – far from it. Here’s the comments (and my comments on the comments) from our reviewer Nils Maier:

This version has been approved for the public. (Good)

The pinch-to-zoom feature seems to work well enough. (Good. But again, not perfect. Some pages just don’t seem to like to get their content scaled, like Facebook’s login screen, so sometimes it looks wacky)

The double-tab (smart zoom) one doesn’t really work for me. It zooms the wrong area, usually truncated. In my tests I browsed a couple of different pages, e.g. http://www.heise.de/newsticker/meldung/Dive-Die-Virtual-Reality-Brille-mit-dem-Smartphone-1854860.html, and tried double tab with the different text columns, but always ended up with a zoomed area not covering the whole text area, e.g. http://i.imgur.com/82ReLBZ.png after tapping into the text paragraph that is mostly visible.
Since the double-tap stuff seems to be experimental still and requires using a Nightly on OSX, I decided to let this slide for now. But please address this in a future update.
(Fair point. We didn’t implement double-tap so that it would zoom in to fit exactly the element that was being zoomed-in on. It was solely a matter of zooming either to max zoom level (if current zoom level is below a certain threshold), or back to 100% zoom (if above that threshold), centered on whatever the mouse is pointing at. Seems fair though that users would expect what they double-tap on to fit to the size of the window.)

 

Design Day came by a week later. It was nice to have an easy-to-demonstrate project that was very visual (“Here, look at this” *twist fingers* *image rotates* “Oh, cool!”). I guess that Strategic Partners presentation was a good de-facto practice run for our Design Day presentation to the panel of judges (not that I still wasn’t completely terrified, but to a lesser extent than I would have been without the practice run). The beginning was easy – we just watched our project video with them(here’s the link to it – feel free to skip past the part from 4:30 to 5:15 - http://capstone.cse.msu.edu/2013-01/projects/mozilla/project-video.mp4) but then came the Q&A session. I think I did well enough on it though to prove my knowledge/ability, except for one time when I got asked 2 questions in a row. I answered the first one and then TOTALLY forgot what the 2nd question was. Now the “correct” thing to do here was to just ask “I’m sorry, what was that other question again?”, but attempting to save myself from admitting I forgot, I used a much better solution – stare into space for approximately 8 seconds. But I did indeed remember it, and I answered it well too (the differences between our 2 zoom methods – fast zoom using a screenshot and pretty zoom using CSS to scale everything).

I thought the day went pretty well for Team Mozilla, and apparently the judges thought so too: we ended up winning the Chrysler LLC Praxis Award for the most technically challenging project (picture: http://capstone.cse.msu.edu/2013-01/slide-gallery/?starting-slide-number=266). I get the feeling that the judges wouldn’t allow the same team to get multiple awards, and that’s too bad because I think we could have at least been a contender for best project video also (Bill gets credit for that – good idea, outline, editing, final product). Nevertheless, I was happy to bring home some hardware (plus, it’ll make a good stabbing weapon should I ever need it for that purpose).

So, overall, good Capstone experience for the 2013 spring semester at Michigan State. I could end this post right here, but I have to mention the post-Design-Day experience of Friday, April 26, 2013. See, we went to the East Lansing bars afterward and I drank for the first time ever. This led me to the following revelation: I will never understand people.

I will never understand why anyone would rather pay $3 for a 2-ounce cup of liquid that (a) tastes awful, and (b) doesn’t come with free refills, instead of paying $1.50 for a regular-sized cup of soda that may as well be huge since you get unlimited refills. Needless to say, with the exception of 4 shots throughout the night, Sprite was my main drink of choice. But, I digress. I still got quite a lot of enjoyment in watching all my fellow Capstone students, normally the smartest group of people I’ve ever been with, just gradually go completely insane as the night went on. Plus, the Tigers won by 10 runs…

Now, detour aside, back to Mozilla. It was quite the learning experience discovering how a web browser works (and this project only covered a tiny portion of Firefox as a whole), and at the end, I’m glad that I’ll get to say I contributed to something that so many people use everyday. It’ll be cool if I see someone using one of the gestures and then I can mention “Oh yeah, I was on the team that implemented that.” “Really, on Firefox?” “Yeah, Firefox.”

So, thanks to my team members Brandon and Bill, as well as the Mozilla contacts Jared and Josh. It was a very long, frustrating, tiring, exhausting project that was also occasionally exhilerating when something I implemented WORKED!!! I’m happy that it ends with a product that can be used by other people. The experience was much appreciated.

Choose your weapon…

Our Firefox add-on has 2 different methods of zooming in or out on a web page. One is simple enough – just scale the content to make it bigger (Done by applying CSS to the document element). It looks nice – animation and video keeps playing, the text looks sharp as it gets bigger. This works really well on say, Google’s homepage. But alas, many webpages are quite a bit more complex, with pictures/menus/advertisements/more text/embedded videos, etc. Using this version of zoom tends to get REALLY slow the more stuff there is on a page – like pinch-in-and-out-then-approximately-4-or-5-seconds-later-the-zooming-finally-changes-direction slow. So… We have a second way to zoom instead.

By default, the zooming method is done via a canvas. What happens is that a snapshot of the current view of the window is taken (done by using the drawWindow() function to draw into a canvas). This canvas is then set to be the background image of a box element, which can be drawn on top of the web page’s content and scaled and moved around. Great – now it doesn’t matter how complex the page is – just take a picture of it and only a single image has to be scaled. The performance is noticeably faster, but there’s a drawback – text gets blurred/pixelly when zooming in, plus animations and videos’ pictures pause temporarily (Although this would be handy if your Spacebar button stopped working).

It still gets a bit more complicated. When zooming in, no problem – all the content needed to be displayed is already inside that screenshot. But when zooming out, some content outside of the current viewport of the window needs to be squeezed in as well so the user can see what they’re zooming out to. Unfortunately, taking a screenshot of the whole page can be a costly operation. Some of these pages seem to go on forever, and since this is supposed to be the “fast” option available to users, it’s best to only take a screenshot of what needs to be displayed. Then we run into another problem – there’s no way to know in advance how far back the user plans on zooming out. There IS a minimum allowable zoom value though – it would appear that the amount of content to take a screenshot of could be calculated as what would appear if the user zoomed out as far as they could – but there’s another problem with that (problems are one thing we never seem to run low on). The user can move the mouse (assuming they have both a mouse and a trackpad) while they’re zooming in to pan up or down or across the page while scaling, so theoretically any area could be reached.

The solution that was come up with was to dynamically add in more zoom canvases and boxes, each the size of the original window, when necessary. So, if say the user zooms out beyond the point that was already captured via a screenshot, add in a new canvas and zoom box from that side and start displaying that also. This way, new content area is only captured when it’s needed.

Ready for one more problem? For some absolutely insane, uncomprehensible reason, the box element containing this canvas image suddenly resets itself back to 100% zoom if:

- it’s on the MacBook Pro or Windows Surface (doesn’t seem to occur on the desktop iMac)

- AND the zoom level is about 2.014 (or so) times its initial value from when the user started pinching (I know, makes sense since it’s a round number. I thought it was exactly 200% at first, but on closer examination, not quite)

- AND the window is wide enough, like about more than 3/4 of the screen’s width. WHY is it only a problem when the window is above a certain width? No idea…

Now this causes a rather ugly jump because suddenly the zoom level is back down to about half of what it just was less than a second ago. Let go of the trackpad and – hey – back to the correct zoom level. We’ve been wondering why it’s been doing this for quite some time now.

I put in a temporary workaround fix – create another canvas already zoomed in to 200% and only draw this on top when zoomed in past 200%. This hides the jumpy canvas zoombox since this newer, more HD zoombox (focused in on a smaller region) is only at its original size, not double its original size – BUT (here we go again) this only works if the user doesn’t move the mouse. Probably okay on the Macbook Pro where there’s only a trackpad and the mouse freezes in place while pinching in and out, but on the Surface the mouse coordinate is the midpoint between the user’s 2 fingers when pinching. This MOVES. It’s noticeable since the HD canvas cuts off the rest of the document.

So – new solution – remove this HD canvas if the mouse moves. Great – except then we just undid our first solution, so it’s back to the original problem – the ugly canvas-resets bug. Sooner or later we’ll have to find the cause and permanent fix to this problem, but until then – keep the window’s width narrow enough when demo-ing our project so it doesn’t occur.

That’s about the gist of the canvas zoom. Faster, yes. Blurrier, yeah, but not to a huge extent. The text is still readable, so it should suffice. The user has their choice of zooming methods, so they can pick the one that works best. Assuming they know more than one option exists, of course…

Never assume anything

It’s amazing how long it can take to complete certain features. I had assumed the incredibly difficult parts of implementing zoom gestures would be:

1) Just recognizing that there’s been a zoom gesture.

2) Taking a screenshot of the webpage.

3) The “easy” part: add in animation to scale this screenshot image in and out and determine how much to shift the page over so it lines up correctly. Just draw it out on paper and use geometry to calculate everything. Simple, right?

Turns out: Noooooooooooope.

Not to say that gesture recognition or taking a screenshot was trivial – far from it (Well, maybe the recognition for twist and pinch – that detection was already included as a selectable condition when we wrote our code. Double-tap recognition was not, but detecting that wasn’t my task, so for all intents and purposes I didn’t have to worry much about it). But I can’t believe how hard this animation’s getting to be to accomplish. Calculating a new zoom level was simple enough – just base it on the amount the user’s fingers move – but there’s a WHOLE lot of other factors to compensate for when calculating scroll bars and translation values that aren’t immediately obvious, namely:

- user zooms out past the edge of a page. The scrollbars can’t be set to negative values or values past the maximum allowed based on the current zoom scale, so translating the entire document is the alternative. Needless to say , the equations get messy quickly.

- user zooms in or out by hand (from the View menu or by hitting command – plus or command – minus). This results in a full zoom, which operates independently from our smooth scaling zoom, but nevertheless affects the original content size.

- whether or not the user added in a toolbar or sidebar, thus shrinking the content area or the window.

- if the user zooms in past the maximum allowable amount, a smooth transition must “bounce” it back to the maximum level. Unfortunately, this changes the scrollbar values as well as the zoom level. Problem is, the scrollbars don’t transition. Again, it must be compensated for by translating the entire document, which makes for rather complex equations.

- The scrollbars have a thickness to them which must be compensated for – IF they’re currently displaying in the window. Should they disappear while scaling, there’s an extra 15 pixels suddenly added to the content area.

- If the mouse was over an advertisement, the web page coordinates got thrown off. Solution: use screen coordinates and the window’s edge coordinates and inital scrollbar values instead.

- At first, the focused window was used to get values like the height and width of the page and various other things. Problem: user selects, say, the Google search bar and then does a zoom gesture – HEY, we got a window of height 30 pixels! Wonder why the vertical position of the page just drastically changed…  SOLUTION: okay, this one was simple. Use the content area of the window rather than the window itself.

- Sometimes, content gets cut off or overflows the screen for what appears to be no reason (typically due to scrollbars being set past their maximum value.or something like that). These tend to be the REALLLLLY frustrating things to find the cause of and solution to.

- plus the length and complexity of equations makes it easy to miss something or add in something extra which shouldn’t be there.

Now if any of these things don’t get factored in to the equations, there’s ALWAYS a scenario that comes up which results in a horrible-looking jump. Fixing these equations to compensate for everything has been my main task on this project for – oh, the better part of, say, 70% of the semester thus far, but it keeps me busy every day. But it’s satisfying every time something gets up and running correctly after a few hours’ work. Slowly but surely these errors are getting eliminated. Unfortunately, slowly but surely the date of April 26, 2013 keeps getting closer and closer as well.

Wow…

I didn’t realize it’s been over a month since last posting an update on the project. So, with my procrastination at studying for tomorrow’s Databases exam acting up, I guess now’s as good a time as any.

We’ve just finished our Beta presentation, which I’d estimate went about 85% well. A couple of miscues on an animation bug and the glare off the document camera which made it difficult to see the screen were the issues, but other than that, I’m happy with the results. It’s nice presenting on the first day so I can just relax watching the next 9 groups present in the next couple of weeks without having to worry about our presentation anymore.

For the most part, our project is starting to look like an add-on: we’ve got our double-tap recognition and animation in now to become feature-complete. But you’ll notice I said “for the most part”, not “completely”, because it’s a tad buggy. We’ve still got glitches in our animation like failing to reset zoom levels when changing to a new webpage, failing to compensate for sidebars that take up horizontal space, and sometimes (this is the REALLLLY annoying one since it’s hard to pinpoint the source) the transition animation when scaling past the edge or past a minimum or maximum zoom level just goes wacky. But the biggest issue right now occurs on the MacBook Pro, where if the window is wide enough, the canvas for some insane reason temporarily resets its zoom level back down after the current zoom scale gets above a certain threshold (like 201% or something like that). But just hold your fingers down on the trackpad for a second, and it fixes itself – AFTER a big jump occurs on screen, which looks really bad. I’ve got an idea for a band-aid work-around, but I’d much rather just not have this problem in the first place.

Luckily, the add-on does what would be expected probably 5 times out of 6. It’s a good thing that most zoom-ins are likely toward the center of a page that don’t require transitioning afterward – those seem to work correctly every time. Still, it doesn’t matter how much testing we’ve done – I was nevertheless terrified during our Beta presentation that we’ve have a major animation jump. Only occurred once – and while the document camera displayed a big glare on the screen, so it wasn’t very noticeable (thank you, glare).

Our project video will be shot later this week. Bill’s got our outline/script written, so I’m going to have to start practicing my lines in the next couple of days. Slowly, it’s all starting to come together for our project. Let’s hope no Murphys-law-esque train-wreck of a disaster occurs in the final month.

Stitch it together

It’s been a while, so I guess now’s a good time for an update post on the project.

We’ve long since solved the problem of getting the scaling screenshot to display on top of the webpage’s content – our canvas is now simply a background element of an hbox which scales itself. For those that don’t understand the technical term or what it does, don’t worry, you’re not alone – I’m not sure I do either. I wasn’t the one who implemented it. But it worked.

The next thing that got taken care of (accomplished by me – I have to state that so I get the credit) was to focus the zoom on where the mouse cursor is. I had to draw everything out on paper but through a couple of hours of trial and error, I got the equations right. Now whatever pixel the mouse is pointing at initially, that pixel stays in place while the rest of the screenshot scales around it.

Now, the more recent issue was drawing outer content when a user zooms out. Before, we’d just take a screenshot of what the window currently looks like and scale that image in and out. Works great – when zooming in – because then all the content to zoom in on has already been captured by the screenshot. Unfortunately, when zooming out, you’d expect the content outside of the current viewing window to get into the picture as well. Unfortunately, just the original screenshot getting smaller and smaller leaves a big outer background to display behind it – like the webpage got cut off. So, the surrounding content should be captured as well so the user can that part too. Now the obvious thing to do is to just take a screenshot of the entire webpage and zoom that in and out, but some of these webpages are huge, and that can start to take up a lot of memory and really slow things down when you do that every time the user zooms out. The problem is that you only want to take a screenshot of what you eventually need to display, but it isn’t known in advance how much the user is going to zoom in or out. So – it’s gotta be done dynamically.

Now taking a new screenshot is an expensive task which can consume time. You can do it occasionally without being noticably slow, but take a new screenshot of the correct area 60 frames per second and the user will start chucking the Mac out the window because it’s taking too long. So, new screenshots the size of the original window but translated away from center are only added whenever the user zooms out enough that the outside edge of the current screenshots make it inside the window frame. If the total composite stitched(hence the title of this blog post) image already fills up the window, no need to create anything new – just scale what screenshots there already are (that’s cheap – can be done quickly 60 frames per second). Tested it myself, and its performance is surprisingly good.

Still on the agenda: getting double-tap implemented (we just found out yesterday the detection of this gesture isn’t auto-supported on Mac like pinch, swipe, and twist were – isn’t that fantastic…), and getting the page re-rendered correctly in the same layout as before – NO WORD WRAPPING! One of those things that you hope there’s a quick and easy solution to but you know there’s no way that’s going to happen.

We’ve got our Alpha presentation Monday. Despite having to use the progress of our project as it was 2 weeks ago, it should be sufficient for displaying our rough initial features to everyone else.

By golly, we’re getting somewhere…

Well, the day started out with another abandon-my-attempt-at-implementing-something-because-it-already-got-it-taken-care-of-without-me moment, but beyond that, I was happy with my contributions today.

A scaling screenshot on pinch-to-zoom has been implemented (no thanks to myself), but for some absolutely insane reason, sometimes that screenshot gets drawn over the webpage’s content, sometimes not (although you can see animation through the “holes” of a webpage where they put advertisement boxes). Michigan State’s website scales fine, mxr.mozilla.org scales fine, Yahoo! sometimes scales fine DEPENDING ON WHICH CURRENT EVENT STORY IS CURRENTLY BEING DISPLAYED!!!, and then there’s websites that don’t hide the content and cover up the animation, like Amazon and Google and our CSE498 page on Michigan State’s website. Weird, I know.

But anyway, back to what I did. Figuring out some algorithms for x- and y- offset, I was able to get zooming relative to the center of the screen (adjusting for scrolling) instead of from the top-left corner. Plus I also got the animation to slow down when getting above or below a maximum or minimum zoom threshold (like it’s offering resistance), then snap back to that minimum or maximum once the gesture terminates. And the individual tabs keep track of their own zoom scale when you switch back and forth between tabs, which was causing a lot of jumping earlier.

Still got a lot of problems to work out – there’s some initial flickering when the screenshot is being created, sometimes it’s not perfectly lined up, we should probably center the zoom on the cursor rather than the screen’s center, and oh yeah – it doesn’t work at all on some websites, but we’re making progress nonetheless.

On the schedule for CSE498 tomorrow – sit back and relax while the final 3 groups give their project plan presentations while being glad that we already did ours a week ago. Maybe I should bring some popcorn to class or something…

We got some momentum going now…

…both figuratively and literally, that is.

Our team was in rather good spirits when Brandon found (via Google) a Gecko event function called DrawWindow() that will help tremendously for zooming in and out. But the thing I liked about today was how I managed to implement an additional feature in rotating images in synthetic documents. Before, when letting go of the trackpad, an image would snap to its nearest 90-degree rotation. That didn’t look right when one just quickly spins an image but doesn’t quite get it past a 45-degree turn, and then it snaps back to the way it was. But now I made our rotate functions take note of how much the user’s fingers moved (and which direction) RIGHT before they let go. Now if there’s any “momentum” on a rotating image, it will spin all the way to the next 90-degree rotation(it’s a lot easier to show it rather than explain it).

But of course, it couldn’t just end there. I had to have my obligatory something-goes-wrong moment before pushing this newest version to our Git repo. Apparently you have to “add” a newly-changed file before being able to commit it (which, thanks to using SVN before, I assumed was only necessary when adding an entirely new file to the repository that didn’t exist previously). Somehow I didn’t have that problem until now. Oh, well. I’m happy with the result nonetheless.