Insights and discoveries
from deep in the weeds
Outsharked

Wednesday, December 21, 2011

Panasonic UB94 wireless adapter driver

Got a new Panasonic flat screen TV for Christmas from my wife (she must really love me!!) and it came with this little USB wireless Ethernet adapter, I guess so they can say the TV supports wireless networks. Like any self-respecting nerd I have a wired network in my house and a switch underneath my TV so of course I don't need this. But do I look like the guy that's NOT going to figure out how I can use it on a computer?

It was a little tricky to track down drivers that work with this so I thought I'd share for future googlers. The device reports itself as a "UB94" when you plug it in. I figured out from some sketchy "driver download"/spam sites that it has an Atheros 7010 chipset, which supports 802.11n. Hey, an upgrade from my old G adapter!

 Atheros doesn't seem to provide reference drivers directly to the public, unfortunately. Some more searching revealed this chipset is shared by the Netgear WNA1100, for which drivers can be downloaded from Netgear. Probably many other devices as well.

To get this thing to work on Windows 7 follow these steps. In the interest in not infringing on anyone's copyright I'll just tell you what to edit rather than posting a driver inf file.

  1. Download & install drivers from Netgear for the WNA1100.

  2. Locate the driver info file, probably:

    C:\Program Files (x86)\NETGEAR\WNA1100\Driver\WIN764\netathurx.inf

  3. Under the [Manufacturer] section, add one line:

    %PANASONIC%   = Panasonic, NTamd64
    

  4. Add a new section after the section for [VERIZON.NTamd64] (actually it probably doesn't matter where you add this, but this seemed as good a place as any)

    [PANASONIC.NTamd64]
    ; DisplayName                 Section                 DeviceID
    ; -----------                 -------                 --------
    %PANASONIC.DeviceDesc.7010% = ATHER_DEV_7010.ndi,     USB\VID_04DA&PID_3904
    

  5. At the very end in the [Strings] section add this line:

    PANASONIC.DeviceDesc.7010    = "Panasonic UB94 USB Adapter"

    This is the text that will appear in device manager. Feel free to personalize.


After that, I just went to the broken device in Device Manager and updated the driver, pointing it to the folder above. If you want you can un-install all the Netgear software and just keep the three driver files -- that is all that's needed.

There, you just saved $19.99!

Tuesday, November 29, 2011

CsQuery 1.0 is imminent

In the last four months I've done a lot of work on CsQuery - on github - a C# jQuery port. I have been using it extensively in a few web site projects and it's quite solid. I've ported most of the jQuery tests that are relevant (dom manipulation, traversing, selection, attributes, utility functions).

Rather than update the list of implemented methods, I've compiled a list of the methods that still remain to be implemented. There are not many. :) Everything else that's not in CsQuery already is browser-DOM specific (e.g. related to events, callbacks, etc.) or is a utility function that I don't think is useful in C#.

jQuery Methods NOT Implemented In CsQuery

        Detach
        Empty
        NextAll
        NextUntil
        End
        WrapAll
        WrapInner
        ParentsUntil
        NextUntil
        OffsetParent
        PrevAll
        PrevUntil
        Prepend
        PrependTo
        Slice
        jquery.Contains
        jquery.Grep

.. plus a few CSS selectors. Additionally, there is extensive support for dynamic/Expando objects using a special JsObject class, and CsQuery.Extend (which works pretty much as you would expect). Though anything that implements IDictionary<string,object> can be used as the target for object creation methods. This lets you work with objects in JSON form, or dynamic objects, almost seamlessly, e.g.:
// Create a new dom from a string of html

var myDom = CsQuery.Create(html);

// "AttrSet" and "CssSet" are the same as Attr(object) and Set(object) - since in C# we can't
//  overload return types. Attr(string) and Css(string) return the values of named items in 
//  CsQuery. This convention is used for methods that can be passed a string of JSON data.

myDom.Select("div.sidebar")
    .AddClass("courier")
    .CssSet("{'border': '1px solid black', 'font-weight': 'bold'}");

// create a new anonymous object. You can also use any conventional object or expando object
// as a source parameter in CsQuery.Extend

var data = new { pageName="My Home Page", url="/myhomepage.html"};

// "null" below is a convention for the empty object {}. You can also pass a new expando object,
// this isjust shorthand. The parameters match jQuery.extend. This merges the properties of data,
// and the object created from the JSON string passed. There's also a CsQuery.ParseJSON method 
// for explicitly creating a new expando object from JSON. Finally, CsQuery.Extend will work
// with conventional objects as the target (first parameter). In this case, it will only update
// existing properties with the new data, since you can't add properties to an existing non-expando
// object.

dynamic dataExtended = CsQuery.Extend(null,data,"{ 'access':'all' }");
myDom.Data("page",dataExtended).Hide();
myDom.RenderSelection();

// outputs: 
//   <div class="sidebar courier" style="border: 1px solid black; font-weight: bold; display: none;"
//       data-page='{"pageName": "My Home Page", "url": "/myhomepage.html", "access": "all" }'>
//   </div>

There are still some other features I want to implement, but I am hoping to get some examples together and create a version 1.0 distribution in the next month or so. The code is solid and well tested, and it makes server-side HTML management a joy compared to WebControls, Razor/HTML helpers, and so on, where you have limited control over server-side HTML layout. And your brain can work with HTML exactly the same way on the server as it needs to on the client. Your whole browser DOM is right in front of you. It's great for scraping too.

I have not done extensive performance testing, but have done a little. It's easily fast enough for real-time HTML parsing. Of course, if you plan to use it on something serving a thousand pages a second, this might matter, but I suspect most people would find it plenty fast. On my laptop, it can parse a 5 megabyte HTML file with over 100,000 unique nodes (the entire HTML 5 spec) into an indexed DOM in 2.5 seconds. Selecting all the DIVs (over 3,300) takes less than 1/100th of a second. Now - 2.5 seconds is an eternity for a web server, but this is meant to be an unrealistic situation, and there would be little reason to parse a big page of static HTML that you had no intention of manipulating. A web page that's 20K, which is more typical, would be less than 1/100th of a second. There's definitely room to make it faster, too, but it's plenty fast now, and I suspect it's a lot faster than manipulating and rendering a page with something like WebControls anyway.

Features that I still want to add:

  • Asynchronous HTTP gets - right now when using CsQuery.Server().CreateFromUrl() to load a DOM from the web, code execution is blocked while the get is performed. This is probably fine for some basic web scraping, but will slow things down a lot for any substantive real-time usage. I started coding for an async model but have not finished yet.
  • Form postback management - there's a basic tool for repopulating form elements from their postback data in the Server() module. This needs to be fleshed out and tested a bit, though, because I have not used it too much as I haven't created a lot of conventional HTML forms lately.
  • Framework and view engine - I've developed a useful, simple framework as part of one project. This includes some custom HTML tags like <csq-include src="..." />, <csq-when [conditions]>...</csq-when> to do things like server-side includes, environment-specific includes, and so on. These are not really specific to CsQuery but rather CsQuery is used to implement them, and they make working with pure HTML a lot easier.
  • Templates - something like the jQuery template plugin. Of course it's a piece of cake to write CsQuery code to do simple substitutions, but it would be nice to integrate some of that functionality into a framework.
  • Client script communication - one of the things that CsQuery makes very convenient is preconfiguring data for client-side controls. For example, say you have a grid control. A typical usage might be to initialize the control with an ajax request upon first page load. This causes the page to be rendered with no data at first, then perhaps an ajax loader shown to the user while it gets the default data. Why not pass the first batch of data directly to the control? It's easy to use CsQuery.Data() to pass data as an attribute of an HTML element, then in your javascript, just grab it with jQuery.Data(). This requires using some HTML element as a payload container. Not a big deal, but I would like to standardize this convention and create methods to abstract it.

Anyway, it's getting close, but feel free to download the project from github and give it a try. The basic usage could not be simpler.

var myDom = CsQuery.Create(htmlString);
var content = myDom["#maincontent > div.title"];
var newContent = myDom["Hello world!"].Css("font-weight","bold");
content.Append(newContent);
Response.Write(content.Render());

Thursday, November 10, 2011

How to run NUnit tests in Visual Studio 2010/MSTest

There are probably millions of lines of test code written against NUnit, and most people take no joy in switching to MSTest just so they can use Visual Studio's IDE. There is an extension called Visual NUnit which adds some support. This is actually a really nice, solid extension, but unfortunately, it doesn't solve the most basic problem: being able to debug tests from directly within your project.

MSTest is unfortunately closed, sealed, locked. It's virtually impossible to extend it. But there is a quick and dirty way to get around this problem and have the best of both worlds: NUnit as a testing framework, but still have the ability to run (and debug!) tests from within VS. And you don't have to sacrifice anything - they will still work in any NUnit test runner.

Step 1: Assert Your Independence

Add both testing framework namespaces:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using NUnit.Framework;

Create aliases in your namespace declarations:

using Assert = NUnit.Framework.Assert;
using CollectionAssert = NUnit.Framework.CollectionAssert;
using StringAssert = NUnit.Framework.StringAssert;

This causes the Assert references to unambiguously refer to the NUnit version. That covers the objects, then there are all the attributes used to mark things for the framework. Luckily, most attributes do not conflict. Description is an exception, you'll have to pick a framework if you use this attribute, e.g. :

using Description = NUnit.Framework.DescriptionAttribute;

will cause all appearances of Description to be recognized only in NUnit.

Step 2: Search & Replace

You need to add all the corresponding MSTest attributes to get the IDE runner to recognize things. Just add both attributes to each class/method, e.g. [TestClass,TestFixture]

FunctionNUnitMicrosoft

Run tests in this class at start (class-level attribute)

[SetupFixture]

[AssemblyInitialize]
[AssemblyCleanup]

(SetupFixture & AssemblyIntialize/AssemblyCleanup work differently - in NUnit a class is marked as [SetupFixture] and has [Setup] and [TearDown] methods. In MS, these just apply to any static methods, with a limit of one each per namespace)

Run once at start per class

[TestFixtureSetUp]

[ClassInitialize]

Run once at end per class

[TestFixtureTearDown]

[ClassCleanup]

Identify a class containing tests

[TestFixture]

[TestClass]

Run before each test

[Setup]

[TestInitialize]

Run after each test

[Cleanup]

[TestCleanup]

... and, of course, A Test

[Test]

[TestMethod]

Step 3: Instance Setup/Teardown

There are some other differences. The setup types for MS must all be static methods, whereas NUnit allows them to also be instance methods. Recoding everything to use static methods is a headache, so I just do this instead. Chances are, your units tests already inherit from some other class. If so, just change your template class. If not, add one. To fake the NUnit instance startup/teardown methods, just use the constructor and destructor of your base class:

public class Test() 
{
    public Test()
    {
        Setup();
    }
    ~Test() {
        TearDown();
    }

    public virtual void Setup()
    {    }

    public virtual void TearDown()
    {    }
}
I've basically just skipped out on using any of the framework class-level setup/teardown methods, and use the regular class constructor/destructor instead. In each unit test, you just override Setup and Teardown. You probably do this already if your tests inherit from a base class, this just changes the mechanism by which they are invoked. I haven't thought too much about possible side effects of this, but it would seem to be functionally equivalent.

Step 4: TestContext

If you happen to be using TestContext, this will be another conflict, since both frameworks have a same-named object. The MS static intitialization methods have it as a parameter, too, whereas for the NUnit framwork, it's a static object you can always access. An easy solution is just to alias the MS one, since you probably haven't written any code against it yet, e.g.:

using MsTestContext = Microsoft.VisualStudio.TestTools.UnitTesting.TestContext;
using TestContext = NUnit.Framework.TestContext;

Now, if you actually want to use the MS static methods, you can just use MsTestContext as a parameter, and TestContext will refer unambiguously to the NUnit one.

Step 5: Convert to Test Project

Visual Studio won't give you the testing tools until you add this to the .csproj file of your test project. It goes under Project/PropertyGroup.

<ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

Step 6. Develop, Test, Not Necessarily In That Order!


You are now done. If you've been careful, nothing you've done will in any way break this when running under NUnit, and all these tests will now run directly in the Visual Studio IDE as well.


In Summary:

  • Alias conflicting objects
  • Use both the NUnit & MS attributes on each class/method as appropriate
  • Deal with non-implemented instance setup & teardown methods using constructor/destructor
  • Convert to a test proejct

... and you should be good to go. While it may take a little bit of work to update large existing test suites, it's mostly search and replace. For new work, just build this into your template, and it's already done.


Tuesday, November 8, 2011

IE7 & quirks removes trailing space from empty HTML elements

Pull up this bad boy in IE7 standards or quirks mode.

http://jsfiddle.net/QNs8s/8/

Using `innerText` or `innerHTML` causes the space after an element to be erased, e.g. if you take

"this is some <span id="field"></span> inline text"

and apply innerText to that span, you get

"this is some  <span id="field">more</span>inline text"

which renders as

this is some moreinline text"

The solution is to start with something inside the span, e.g.

"this is some  <span id="field">&nbsp;</span>inline text"

I can't believe I've never come across this before, but googling didn't turn up anything about it. Another irritation for supporting old IE.

Thursday, October 13, 2011

Another new ImageMapster feature: Area Zoom

A couple months ago I added the ability to dynamically resize image maps. This made a lot of other things possible, and I've just gotten around to integrating one of them into the codebase. (And I should also mention that the codebase just went over 100K and I will probably fragment out the major features for release 1.3 so you can build a version only with the features you need).

The new Zoom feature lets you zoom on a specific area. It's pretty limited in terms of features/flexibility right now but without too much code you can do some neat stuff without too much code. Here's a demo I whipped up to let the map "follow" you to zoom in on areas, which could be very useful in presenting a user with an image map that contained some large, but some very small areas.

Fiddle with it

Hover over any area for a second to see it zoom. (If this demo doesn't work, open this post alone -- the include script could conflict with the one from the earlier resize demo)

There are some issue to be worked out. It's not too hard to make things go completely haywire by mousing all over the place - mouse positioning data is screwed up in blogger, not sure why. The style when zooming shifts things a little bit. But the basic functionality is there.

































































Friday, October 7, 2011

HTML5, Firefox, canvases, oh my.

There's been a bit of a surge in feedback about ImageMapster in the last few weeks, which is fantastic because it means people have been using it. At the same time, it has made me acutely aware of the complexities of writing software that attempts to abstract a difficult problem into a general-purpose solution.

The problem I'm trying to solve is simple. Take an image. Create some rules about what happens when the user interacts with it. That's not that hard. They did it with Pong in 1972. Of course, Pong only had to work on one platform. Actually, the platform it worked on was designed only for pong.

My problem is a little more complicated. This plugin has to work on a whole bunch of different web browsers. Each of these browsers has a whole bunch of different versions.

In modern browsers, the software uses the HTML5 <canvas> feature to create its effects. HTML5 is an emerging specification. While this particular feature is well defined and has been available in common browsers longer than most, it's also one of the more sophisticated capabilities of a browser. It's like having your very own mini-Photoshop running in a browser. You can really do all kinds of crazy stuff.

Imagine if two different companies were trying to create Adobe Photoshop from a set of specifications. What are the chances that they would work the same way? With web browsers, though, it's about a half-dozen completely independent code bases.

Of course, canvases aren't nearly as complicated as a full design package, but there's still a lot of room for nuances. And since the spec is emerging, and has been implemented in various browser versions with varying degrees of success for a few years. Ideally, you want to code in such a way that avoids bugs that may exist in certain implementations.

I recently fixed a bug that appeared with the release of Firefox 6. It had to do with masks. One of the features of ImageMapster is the ability to create hotspots in images, but also create an exclusion area. In the picture at left, the circle in the middle of Texas is a mask. (Technically, Texas is the mask, since it allows the hole to show through, but I thought it would be awkward to refer to the transparent areas as "holes"). In this configuration when someone mouses over Texas, the area in the middle will not get highlighted. If they move the mouse into the mask area, the highlight disappears.

Anyway, after substantively changing the approach used to create the masks to get around the problem with Firefox 6, I found that masks no longer worked in Firefox 3.6!

In old Firefox canvas implementations, there appears to be a problem when using globalCompositeOperation = "source-out" -- a critical feature for rendering masks - to make successive copies of canvases. That is, to create the effect, I would render all the "holes" on one canvas, then render that onto a canvas with the highlighted effect with "source-out", causing the holes to be excluded. Finally, though, I need to render those onto a third canvas (the one that gets displayed) which contains other highlighted effects. Somehow, this three step process completely fails in older Firefoxes.

Previously, I had achieved the effect with a simpler, two-step process. I render all the masks onto a canvas. Then, on the same canvas, I set "source-out" and render the highlighted effect. This works on every browser, except Firefox versions later than 6, which for some reason would do nothing with the masks.

I still haven't managed to isolate the exact nuances of the different behavior in different Firefoxes, because it involves many different actions in sequence on a canvas: paths, clipping, fills, context states, and so on. But suffice it to say, if I want this to work on all known versions of FF (and versions 4 and lower are still not uncommon), I will need two entirely different code paths for this feature.

Then, of course, there's VML, the vector markup language used by Internet Explorer prior to version 9. This is completely different again. In some ways, though, it's refreshingly simple. It works. It doesn't do everything that canvases do, but at least the implementation is consistent across all browsers that use it (that is, Microsoft), and at least it will never change again.

And finally, there's Apple computers. I test this thing on an iPad regularly. I don't own a Apple computer, though. My assumption has been, if it works on Windows Safari, and it works on my iPad, it's probably got a pretty darn good chance of working on apple Safari. I mean, they're all webkit in the first place, and you'd figure that a mobile touch screen device is a pretty good stress test for this kind of software.

No such luck. I got a user reporting no go on his mac this week. Then, another user tells me it doesn't work in chrome of all things! Chrome is what I use 90% of the time. I might forget to run tests in Firefox 3.6 but I never would push something that was broken in chrome. What is going on here? Well, most likely, it's something outside of ImageMapster: a local configuration issue. A firewall problem blocking images fetched with javascript, possibly due to CORS restrictions. But it's hard to troubleshoot, and as far as someone trying to use this can tell, it's just broken.

So what is the point? All this comes down to the whole point of this in the first place. Writing software that does complicated things on the Web and works for (almost) everyone is hard. But if it doesn't work for almost everyone, it's also useless. At the end of the day, it has to work. And this is the real reason why the next generation of the web's evolution has been slow. HTML5 canvases have been in Firefox, Chrome, and Opera for several years now. Yet, few web sites use these features. The big reason is because Internet Explorer versions older than 9 still have a substantial market share, and few people want to try to accomplish such things using VML. But another reason is that the implementations remain inconsistent across browsers, and it's difficult troubleshoot because not that many people are doing it. There's just not that much information out there about these nuances.

But to pass on it entirely because of these problems is to remain stuck in the last decade. There's tremendous power in the modern web browser. I mean, you can play angry birds right in your browser!

That makes my plugin look like child's play. Of course, they make no bones about it working in anything other than Chrome. (It supposedly works in Firefox too, but isn't really fast enough). The future is here - it just waits on the developers to embrace it, warts and all.

Friday, September 23, 2011

ImageMapster 1.2 released

I "finalized" version 1.2 of ImageMapster, which includes a few major new features and a slew of other improvements. Of course, even after waiting two months to officially call it 1.2, I found a couple bugs within hours of releasing it. Such is life. So we're already at 1.2.2 which corrects a couple minor issues on the initial release.

I also finished a significant update to the project web site which I hope will make it a lot easier for new users to understand the plugin, its features, and getting started.

The new version includes many new features (most of which have been in the beta for a month or two):

  • Automatically scale image map data to match the effective image size using scaleMap option
  • resize method will resize an existing, bound imagemap dynamically with visual effects
  • includeKeys can be used to bind staticState areas to active areas and mouse events will affect active areas. That is, if area A is staticState=false and area B is normal, using includeKeys='B' for the area A data will mean that mousing over of clicking area A will cause action on area B.
  • Performance and stability improvements on startup with complex or slow-loading images across all browsers (some edge cases, especially with certain Firefoxes and IE7, didn't bind consistently).

  • ... and lots of other little improvements/tweaks/fixes. See change log on github for everything.