Notes on Symbian, Qt, QJson, DLLs and Capabilities

I'm working on some small application on Symbian-Belle platform. Everything was doing well until I've hit a place where I had to parse a few JSON files. I've found a QJson library, which seemed very relevant and easy to use (although it seems to return everything as QVariantMap, which probably is not that lightweight..).

This library is however not provided in a precompiled form. It is source-only, but the source is openly available at http://gitorious.org/qjson/qjson/. Fortunatelly, it is prepared to compile under QtCreator's Symbian projects.

Initial project setups

After fetching the library from Git, I've tried compiling it and it went well. I wanted the lib to compile along with the main project. I've created a top-level SUBDIRS project and put QJson underneath, and also I added there the project with my application. I've set all inter-project dependencies and corrected the build and run configuration to actually run my application (SUBDIRS top-level project does not set it by default, it has to be manually corrected).

I've chosen the simulator configuration, pressed Run, compilation passed, but linking did not:

Error: file 'qjsond.lib' was not found

The default QtCreator's dependency management added the lines:

win32:CONFIG(release, debug|release): LIBS += -L$$OUT_PWD/../QJson/lib/ -lqjson
else:win32:CONFIG(debug, debug|release): LIBS += -L$$OUT_PWD/../QJson/lib/ -lqjsond
else:symbian: LIBS += -lqjson
else:unix: LIBS += -L$$OUT_PWD/../QJson/lib/ -lqjson

and the QJson library has generated only 'qjson' binary. I am running on Windows, so building for the Simulator looked for the 'qjsond' symbol, just as the second line clearly states. BTW. this is the defacto standard way of naming debug versions of libraries on win32 introduced by Microsoft's VisualStudio series.. Anyways, here's not relevant. The QJson, even in debug-mode, builds only without the -d prefix, so I removed it:

win32:CONFIG(release, debug|release): LIBS += -L$$OUT_PWD/../QJson/lib/ -lqjson
else:win32:CONFIG(debug, debug|release): LIBS += -L$$OUT_PWD/../QJson/lib/ -lqjson
else:symbian: LIBS += -lqjson
else:unix: LIBS += -L$$OUT_PWD/../QJson/lib/ -lqjson

Everything compiled well, linked well, so I run it in the Simulator. And it run! yay.

And on the device..

So, the next part was to run it on the real device - and here another class of problems began. After building, linking, deploying and installing, the IDE told me:

Launch failed: Command answer [command error], 1 values(s) to request: 'C|4|Processes|start|""|"MyApp.exe"|[""]|[]|true'  
#0 {"Code":-46,Format="Failed to create the process (verify that the executable and all required DLLs have been transferred and that the process is not already running) (permission denied)"}  
Error: 'Failed to create the process (verify that the executable and all required DLLs have been transferred and that the process is not already running) (permission denied)' Code: -46

and the launch process instantly was aborted. Of course the application was launching properly back when the json-related code and library was not yet added to the project..

I've checked and the QJson library was indeed properly installed on the device, so it was not about missing files. Also, the application it self has been properly installed too. Anyways, in the parentheses there's a note permission denied which was quite suspicious. When I tried to run the application manually on the device, I've got:

Dostęp do aplikacji został zablokowany przez administratora

in english, I'd be something like:

Access to the application has been blocked by the administrator

actually, on devices with english language set, the message is:

Unable to execute file for security reasons

Well, this was not what I expected when adding a DLL. I expected a crash, SIGSEGV, Kernel-Panic-#32 etc, but not some security/administrative restrictions. I've searched a bit on Nokia'a documentation and I've found a page that mentioned that DLL projects also can have Capabilities specified, just like normal .EXE applications.

QJson's DLL Capabilities and UID3

Actually, bad Capability definitions could have caused that error, so I've checked the .pro files from the library project. In 'src.pro' I've found lines:

#TARGET.UID3 = 
TARGET.CAPABILITY = ReadDeviceData WriteDeviceData

From elsewhere I knew that those capabilities require the application to pass the Symbian-Signed process. Currently my app was just self-signed, as this is the QtCreator's default setting. I didn't want to setup the full signing process yet and my application does not require those capabilities at all, so I commented-out them, rebuild, and it didn't help at all :)

Even worse, after next few attempts, I started getting the same error as in here http://stackoverflow.com/questions/12705282/use-qjson-in-my-qt-symbain-app:

error: Installation failed: 'Failed to overwrite file owned by another package: c:\sys\bin\qjson.dll in (....)

Then I remembered that near those capabs in the qjson library's project file, there was a UID3 commented out! So, probably the build process has been using some temporary or random one, and it has changed! The installer cannot verify that I am in fact updating an old version, and thinks that some new incoming .sis file tries to overwrite a file that does not belong to it.

I've manually uninstalled the old package, updated the qjson project with

TARGET.UID3 = 0xE0123456

then rebuild and the installation succeeded. However, still it could not launch due to the security error, even with the spurious capabilities removed.

DLL Capability explanation

Here's an article that is a must-read if you are ever going to use DLLs on the newest versions of Symbian platform. You see, when I last touched the subject, there was no such thing as 'capabilities' on Symbian, and the other platforms like WP7 define them a bit differently. Anyways, the article is http://www.developer.nokia.com/Community/Wiki/Shared_Library_DLLs_on_Qt_for_Symbian

Inside there's a small section named http://www.developer.nokia.com/Community/Wiki/Shared_Library_DLLs_on_Qt_for_Symbian#Capabilities Set platform security capabilities. In this section a very important phrase is seen:

Capabilities are treated slightly differently for EXES and DLLs. In essence, while an EXE must be granted the capabilities required to call the APIs that it uses (or which are used by its loaded DLLs), a DLL must be given all the capabilities of all the EXEs that might need to load it. For a DLL used by a single application exe this would be the same as the application's capabilities. If however the DLL is to be used by arbitrary clients, you will need to give it as many capabilities as possible.

and also, far later:

Note that the application .pro file should also specify all the capabilities that the EXE needs to use any protected API that it calls or the DLL calls - the DLL should have all the capabilities specified in the EXE (and it may have more).

What does it mean? In EXEs the Capabilities are specifying what that EXE is allowed to try to use. If an EXE tries to read the device IDs, it needs ReadDeviceData, or else an error will be rised. The DLLs in turn are always using the capabs that the calling EXE provides - otherwise it would be dumb or useless in terms of security. Thus, specifying Capabilities on a DLL project does not provide any access rights to the DLL. If a DLL reads device IDs, then the calling EXE must have the Capabilities specified..

However, the Capabilities are also used in DLLs projects, but their meaning is drastically changed: now they specify in what contexts the DLL is safe to be used. They are a means of describing the DLL safety. For example, if a DLL is marked by Capability ReadDeviceData, it could mean for example that it has been checked to not peek some user-critical data and send them over the internet, and therefore this DLL is safe to be used by applications that have ReadDeviceData flag..

All those facts taken together create a simple security rule:

If your application has some Capabilities specified, then all DLLs that are loaded by the application must have at least all those Capabilities.

In the example of my project, my application had specified 'NetworkServices', and the QJson library by default has 'ReadDeviceData WriteDeviceData'. It is a complete mismatch, and when the OS tried to run my application, it discovered that my "network-related application" tried to load a DLL that is "not safe" in terms of networking, so it halted the process..

Solution

The solution is very simple. I've just had to leave my application project as it was, with no changes, and only add the network capability to the QJson's project (to mark it as safe-to-use-with-network):

TARGET.UID3 = 0xE0123456
TARGET.CAPABILITY = ReadDeviceData WriteDeviceData NetworkServices

Please note that I even did not have to remove the *DeviceData capabilities: my application does not have them.

Better fix?

According to the Nokia's document and to simple reasoning, the DLLs should have the widest possible Capabilities assigned, especially if they are to be by different projects for different applications.

Thus, I cannot fanthom why the QJson library comes only with *DeviceData capabilities. Actually, JSON format is heavily related to networking, so it is quite obvious that the application that use this library will have NetworkServices capability!

I'll file a patch to the project with more capabilities added. In the meantime, please remember that if you use a DLL project, you must ensure that the DLL capabs are at least as wide as your EXE's.

Random thoughts on Moq and incremental assertions

This text relates to my earlier proposal of mock.Count method: https://github.com/Moq/moq4/pull/17#issuecomment-6538764

I publish this text here just not to spam the issue tracker with many pages of loose thoughts.

I think that maybe I expressed myself not too clear. I was not saying that Reset is bad/useless/etc in a general manner. I meant that:

• Reset() -- resets whole log - is easy to use
• Reset(filter) -- removes selected entries - has its pros, but despite being similar to Verify, it is much harder (than Verify) to use properly in any non-trivial cases, and in some case, may even be useless while cumulative Verifys would work
• .Count(filter) -- is not as "clean" as Reset(filter) due to needed local variables, but does not have the drawbacks that make Reset(filter) possibly hard to use

If you wonder on drawbacks: it is the state mutation, performed with "greedy" filter.

Facts:

• Verify and Count works intuitively, before they doesn't change anything, and just check or summarize the current state.
• Reset on the other hand modifies the current observation state.
• Filters are built of two types: precise It.Is<X>( x => spec-of-x ), and range-wise: It.IsAny<X>()
• IIRC, even the precise filter can be made range'd, for example with spec: val >= 15

Imagine a case:

interface IManager
{
    void Register(int priority, int itemType);
}

class Tested
{
    ...... ctor, IManager mgr variable, ...

    public void DoWhizz(..params, params..)
    {
        ...blah...
        mgr.Register( 0, 20 );
        ...blah...
        mgr.Register( 0, 20 );
        ...blah...
        mgr.Register( 5, 0 );
        ...blah...
        mgr.Register( 5, 20 );
        ...blah...
        mgr.Register( 5, 20 );
        ...blah...
    }
}

void TheTest()
{
    Mock<IManager> mock = .....;
    var obj = new Tested(mock);

    obj.DoWhizz(.....);

    // // // mock.Verify(o => o.Bar( It.IsAny<int>(),      It.Is<int>(val => val > 15) ), Times.Exact( 4 ) );     // A, would pass
    mock.Verify(o => o.Bar( It.Is<int>(val => val > 3),    It.IsAny<int>()  )         , Times.AtLeast( 3 ) );     // A, passes

    mock.Reset(o => o.Bar( It.IsAny<int>(),            It.Is<int>(val => val > 15) ));   // B

    obj.DoWhizz(.....);

    mock.Verify(o => o.Bar( It.IsAny<int>(),            It.Is<int>(val => val > 15) ), Times.Exact( 4 ) );     // C, passes
    mock.Verify(o => o.Bar( It.Is<int>(val => val > 3),    It.IsAny<int>()  )         , Times.AtLeast( 6 ) );     // D, FAILS
}

This test was written with some black-boxy assumptions:

• during all DoWhizz calls, it should Register at least 3 items with high priority
• during the very first call, we do not care about the item types, as it may for example, reregister some historical things
• during later calls, we want exactly 4 new items of "higher type" to be processed

Line tagged as (A) is obvious. At this point of time the val>15 is ignored, but if I knew that there is no history/etc, I could use it and it would pass.

In line (C) for some reason, I wanted to not accumulate expectations and write 8, for example, for test mainterance reasons.

So, in line (B), I resetted the counter for the calls in question. Maybe I was new to that, or maybe I did not think enough about it, or maybe just my actual case was a bit more complex, the issue nested deeper and harder to trace - whatever. I used the same filter I will use later in C, and also it is all the same filter I'd use in (A) if I could/knew - it "felt obvious and natural".

The effect is that the under-specified reset filter in line B has implicitely mutated counters not only for the first, but also for the second issue that this test checked. I resetted counts for second param, and the count for first param has implicitly changed, and at point (D) it will be now 5, not 6. This is obvious when you look at implementation of the Tested class, and if you know exactly how the invocation log was 'recorded'. Elsewise, this is not so obvious.

To perform proper reset in this case, the reset would have to look like:

    mock.Reset(o => o.Bar( It.Is<int>(val => val <= 3),   It.Is<int>(val => val > 15) ));

and this starts to present the possible complexity of having to manually slice the parameter domain into disjoint regions to be droped. With integers and 1 or 2 params this is easy, but later it will start to be a tough piece to mantain.

Assuming you dont want to analyze and slice the domain, you may try just to "intuitively" reset for the other paremeter too:

    mock.Reset(o => o.Bar( It.IsAny<int>(),                It.Is<int>(val => val > 15) ));
    mock.Reset(o => o.Bar( It.Is<int>(val => val > 3),                It.IsAny<int>() ));

Right? Not! First note: what if there were more parameters? Add more resets? Second note: Actually, with each such line, you would introduce more uncertainity, because the second line has its second param open. Such two lines aggregate to:

    mock.Reset(o => o.Bar( It.IsAny<int>(),    It.IsAny<int>() ));

so, actually, they are unfiltered reset that clears everything related to that method. If it has helped in our case, then we did not need the filter at all. If it did not help - then the reset must be specified properly by slicing the domain and praying that we have few, separable parameters:

    mock.Reset(o => o.Bar( It.Is<int>(val => val <= 3),   It.Is<int>(val => val > 15) )); /// <--- the only safe and proper way

So, while it is useful feature, in my view, in most nontrivial cases, some unpleasant cases show up:

• case1: those not fluent in the topic, will "intuitively" expect that line (D) will have the count of 6 - because they have reset/filtered the OTHER parameter
• case2: those more experienced, will know that clash can happen, and will manage to avoid it, but they will be unable to guess what count they should expect at (D), because the DoWhizz's will be a black box
• case3: the parameters will be too entangled, or clash will be deep enough to make the parameter domain too hard to slice properly, thus making it impossible to determine the count at (D) after the partial reset

As I already said, there's an 'oops' that with Verify & Times, the writer has to specify the absolute bounds, and is unable to express relative bounds. The addition of Reset did not help - it solved one assertion by resetting it to zero, but has damaged the other one by making the absolute value hard to determine -- because there is no way to learn the number of invocations other than asserting it Verify & Times and manually reading the exception text.

If Count were added, the case is trivial to fix:

void TheTest()
{
    Mock<IFoo> mock = .....;
    var obj = new Tested(mock);

    obj.DoWhizz(.....);

    mock.Verify(o => o.Bar( It.Is<int>(val => val > 3), It.IsAny<int>()  )         , Times.AtLeast( 3 ) );

    mock.Reset(o => o.Bar( It.IsAny<int>(),   It.Is<int>(val => val > 15) ));   // <--- damages some counts
    int cnt = mock.Count(o => o.Bar( It.Is<int>(val => val > 3),        It.IsAny<int>()  ));   // just CHECK them

    obj.DoWhizz(.....);

    mock.Verify(o => o.Bar( It.IsAny<int>(),  It.Is<int>(val => val > 15) ), Times.Exact( 4 ) );
    mock.Verify(o => o.Bar( It.Is<int>(val => val > 3),  It.IsAny<int>()  )         , Times.AtLeast( cnt+3 ) ); // and USE them
}

Instead of analyzing what and how to reset and writing several resets to properly clear the parameter domain chunks, I just read the current value and use it in assertion. In my view, it is as short, as easy to comprehend and also as readable as it can be.

The same test with Count only is in my point of view more straightforward, because you dont have to think what hidden sideeffects could the Reset have:

void TheTest()
{
    Mock<IFoo> mock = .....;
    var obj = new Tested(mock);

    obj.DoWhizz(.....);

    mock.Verify(o => o.Bar( It.Is<int>(val => val > 3), It.IsAny<int>()  )         , Times.AtLeast( 3 ) );

    int cnt1 = mock.Count(o => o.Bar( It.IsAny<int>(),  It.Is<int>(val => val > 15) ));
    int cnt2 = mock.Count(o => o.Bar( It.Is<int>(val => val > 3),        It.IsAny<int>()  ));

    obj.DoWhizz(.....);

    mock.Verify(o => o.Bar( It.IsAny<int>(),  It.Is<int>(val => val > 15) ), Times.Exact( cnt1 + 4 ) );
    mock.Verify(o => o.Bar( It.Is<int>(val => val > 3),  It.IsAny<int>()  )         , Times.AtLeast( cnt2 + 3 ) );
}

I hope I managed to show you that:

• Multiple subsequent calls to Count are orthogonal, while calls to Reset are not
• addition of Count overlaps with Reset only partially
• addition of Count helps to deal with some problems of Reset
• Count does not have the problems of reset
• compared to Reset, the Count forces you to add similar numer of "extra lines", usually less (no params slicing)

I wrote a tome, so I'll add a few words more :) The last example shows that Count is used like Verify with some sort of initial checkpoints. It is a little work to write cnt+4, and it may look ugly for some, so I also suggested Times.Add() method for some sugar.

Actually, what I'd like to see is:

void TheTest()
{
    Mock<IFoo> mock = .....;
    var obj = new Tested(mock);

    obj.DoWhizz(.....);

    // var mark1 = mock.Mark(o => o.Bar( It.IsAny<int>(),   It.Is<int>(val => val > 15) ) ); // either that
    var mark1 = mock.Verify(o => o.Bar( It.IsAny<int>(),   It.Is<int>(val => val > 15) ), Times.AcceptAny() );  // or that
    var mark2 = mock.Verify(o => o.Bar( It.Is<int>(val => val > 3),  It.IsAny<int>()  ), Times.AtLeast( 3 ) );

    obj.DoWhizz(.....);

    mark1.VerifyNext( Times.Exact( 4 ) );    // no more filter copying just to verify the same thing
    mark2.VerifyNext( Times.AtLeast( 3 ) );
    mark2.Verify( Times.AtMost( 3 ) ); // why not have a Verify and VerifyNext here?


    // below - contrived features :)

    Assert.True( mark2.Count % 3 == 0 ); // Count returns current last count stored by marker, if the Times is not enough
    mark2.Count = 15; // marks are meant to be simple. You can do that if you know better

    int cnt = mock.Count( mark1.Observed );  // you can always check the most recent without bumping the markers
    var markX = mock.Verify( mark1.Observed, Times.AtMost( 5 ) ); // marker remebers the filter? coooool
}

Currently, verify now returns void. This is a waste. This is a great handy way to provide simple checkpointing that could increase readability or offer new sly features. The API change is minimal: Verify works as always, throws when Times are not satisfied. But when succeeds, returns a marker with some information on the last state of the observation.

However, you'd get a marker only when Verifying something. How about if you just want to have a new marker without checking anything? It would need a new factory method, or can be solved just by adding a pass-all option to times solves it.

The Mark/Checkpoint object may be a simple struct that just remembers the Lambda and the Count. The VerifyNext would update the internal counter, so several VerifyNext would behave intuitively. The Count property is only used as a base offset for "next checks", it does not have anything to do with actual invocation history log - it just offsets the Times during VerifyNext. The Observed property (or Filter, or Condition, name is not so important), carries the original expression provided to Verify. The whole Mark is really a Mark and carries 100% of the initial information about what the writer wanted to observe. This makes making similar assertions much easier.

With such thingies, the example from earlier parts would boil down to:

void TheTest()
{
    Mock<IFoo> mock = .....;
    var obj = new Tested(mock);

    obj.DoWhizz(.....);

    var mark1 = mock.Verify(o => o.Bar( It.IsAny<int>(),   It.Is<int>(val => val > 15) ), Times.Any() );
    var mark2 = mock.Verify(o => o.Bar( It.Is<int>(val => val > 3),  It.IsAny<int>()  ), Times.AtLeast( 3 ) );

    obj.DoWhizz(.....);

    mark1.VerifyNext( Times.Exact( 4 ) );
    mark2.VerifyNext( Times.AtLeast( 3 ) );
}

which in my opinion is far superior to addition of .Reset and .Count

There is no rush, and I am aware that most of the people does not do any "incrementality" in their test, they probably just multiplicate the test cases five times and test things separately. I do not negate this approach: sometimes this is good and desired. However, when you have nontrivial setups to perform, you actually want to either reuse as much as possible (even the assertion builders), or to check as much as possible at the cutpoints that you just meticuously crafted.

Addition of .Count is ready and is an instant. Change is trivial and provides similar features to 'marks', just with less syntax sugar. By throwing the Reset in, we add more sugar, but Count is still needed.

When I get another few days off from work, but if I will try to write it, and probably post it as separate proposal.

Some flashbacks on WP7 Pivot/Panorama/WebBrowser problems

Completely by accident, I just stumbled upon:

http://www.scottlogic.co.uk/blog/colin/2011/11/suppressing-zoom-and-scroll-interactions-in-the-windows-phone-7-browser-control/

Colin - Thanks for credit! And for talking more on the topic. I've never had enough time to describe a more-full solution than that one lenghy post. A user asked me recently about the code, and, of course, I shamefully still lack the time, so I'll give him link to your article.

If somebody is interested in, my original post Colin refers to is: http://stackoverflow.com/a/7347448/717732 and I also recommend to you to read another one http://stackoverflow.com/a/7391698/717732 as it also contains few bits.

At that time, I wrote quite a few posts related to WebBrowser and Pivot/Panorama on WP7, but that one is the most complete - but still, it is not complete.

I really regret I have not written in full about it at that time. I remember I did a lot of research on what is and what is not possible, and how to achieve useful effects. I realize that even at 7.5 it is still an issue, but unfortunatelly, I cannot currently provide you with my original code and ready-to-use components, because .. it is a part of published application ('OnetNews') and the last time I spoke about it, its owner was too happy about such idea of sharing the code..

BTW. I've just remembered a very important note: the WebBrowser control on WP7.0/7.1/7.5 seems to have a serious memory leak, that manifests rarely, but (almost) deterministically if your application meets certain criteria. For example, I've noticed that one of such case is .. putting few WebBrowsers across a few Panorama pages and dynamically changing their URI/Source upon swiping the Panorama's pages. I failed at pinpointing what exactly is the problem. It seems to be some race condition that causes loss of even few megabytes of memory per wipe-and-uri-reload. For a medium-sized application, it means that your application may crash afrer 14-30 page swipes. Also, be careful with images you place or link to - I remember I could easily whole application by navigating the WebBrowser to a page that displayed some large photos (frankly, all images seemed to be GC'ed properly by the WebBrowser - if only the app didn't crash).

If I manage to find time and my old notes, I'll try to post more info, but again I can't promise when it will be :|

VisualStudio 2010 registry chaos

Actually, not a chaos, but it is a catchy word..

The case started with a short note I've received from EmperorXLII:

I wanted to let you know that the new .vsix does not support
command-line execution through MSTest.exe of test assemblies.

In other words, from a VS command prompt,
    mstest /testcontainer:MyTest.dll
reports "No tests to execute".

I still had the .reg file from the previous version; adding
that to the registry fixes the mstest issue.

I immediatelly thought about two registry entries I intentionally skipped in the newest version, but they were really irrelevant. What really suprised me is that after running old .reg file - it started working. It simply could not.

If it started working when he used the old reg file, it means that he probably accidentially activated an version older than 2.1 (older than the one with 'installer'). The MSTest most probably loaded some old modules from the 1.2 or 2.0 version, not deleted and still sitting in the /PrivateAssemblies.

After what I've traced in the last few days, I'm quite sure about that!

Few things about MS QualityTools internals in VS2010

As I was previously searching for the fix for IMessageSink problem, I was tempted also to check, why the runner does not respond properly when commands like "Run tests in current context" or "Run tests in current class" were not handled properly. It was very strange, because looking at the code of the runner, everything seemed in a perfect order. The tests were discovered, then were run, the results were displayed - although in somewhat distorted way: the Owner column was holding the class name, the ClassName column was empty, the FailureCount was not recorded at all, etc.

It was obvious that something was not right. The case of Owner was trivial - even the original author warned that he couldn't get the ClassName filled properly, so he artificially injected the name of the type to the Owner, just to be able to see, group, and sort the tests.

I immediatelly thought, that maybe the Visual Studio is relying on that very column, the ClassName, to navigate between the test and the code, and maybe it is some partial cause why the "Run in current context" does not work.

Obscure bug in xUnit's .Net Remoting layer vs. XUnitForceLegacyCallback

with nods to EmperorXLII and his work for xUnit

and of cource, to the whole xUnit's team, too :)

There's an "unknown incompatibility" problem mentioned in the comments, which forced the original author of "VSTS Test Runner for xUnit" to add the XUnitForceLegacyCallback hack was quite tough to trace. It took me more than 8 hours to diagnose and fix, so if you, dear reader, are here already, please at least skim through it and either remember what IMessageSink really is, or write down somewhere the helpful external links :))

12:50, press Return